Setting up home network with KVM and LXD


#1

Hello,

I’m looking for guidance in setting my home server. I have a HP Proliant Gen 8 Microserver G1610T box upgraded with 12GB RAM.
I was using it and plan to continue using it with some critical applications AND as a homelab to test an play with new OSes, technologies and solutions.

TL;DR
If this post is too long for you, just please look on the diagram and the questions below it.

Background

In short, my plan is to isolate my production environment from testing ground, so I won’t mess up with services that should be up and running most of the time. Also, I am occasionally away from the physical box so if I screw up network configuration (which I’ve done…) or my host won’t boot up I’m f.u.b.a.r.. For that reason, I decided to have one VM for all critical applications and others for testing, according to my needs.

What I want

  1. Separate testing environments from services I don’t want to break
  2. Have remote access to administer machine, install / reinstall OSes
  3. Use snapshots to have point in time backups that I can revert to if something goes wrong

My plan

To achieve further fail-proof isolation and preserve from config contamination, I want to put those critical application in containers, preferably LXD ones because I then can provision them with ansible without much tweaking (the same way I’d done without any layer of vm / container).
Because my server is in my home behind my home router in order to access it from the internet I have to forward ports to it, so I my solution is as follows

  1. Port 80 and 443 for web applications are pointed to a container with nginx serving as reverse proxy (pointing cloud.mydomain.com to container 2, git.mydomain.com to container 3 etc.).
  2. All other applications are separated in LXD containers according to their purpose
  3. Other services (like XMPP) are forwarded (by router directly to specific container
  4. Bare-metal host OS is Ubuntu 18.04 installed on LVM to achieve easy rollback and backup with snapshots. Livepatch and unattended upgrades are setup. Host OS have minimal set of applications installed and all ports except SSH opened. For visualization KVM is used.
  5. On VM PROD another Ubuntu 18.04 is installed with latest snap LXD and several per-service containers.
  6. Both HOST and VM prod have bridge setup so every new VM and container can have its own IP on my LAN so further configuration will be easier.
  7. Each single machine (VM and container) is provisioned by Ansible with separate playbooks.
  8. For snapshots and backups ZFS is used for containers and LVM for VM PROD and HOST
  9. Storage consists of 1 SSD for HOST OS with setup and 4x3TB WD RED drives. I have not decided how to set them.

Below I attach a diagram of what I plan to setup and would like to ear some feedback about it - what are possible pitfalls and drawbacks and what should I do differently.


                                                   .-----------------.
                                                   |      HOST       |
                                                   | (on bare metal) |
                                                   |   __________    |
                                                   |  [_|||||||_°]   |
                                                   |  [_|||||||_°]   |
            .------------------------------------->|  [_|||||||_°]   |-----------------------------------------------------.
            |                                      |-----------------|                                                     |
            |                                      | Ubuntu 18.04    |                                                     |
            |                                      | on LVM          |                                                     |
            |                                      | +br0 bridge     |                                                     |
 .---------------------.                           '-----------------'                                                     v
 |       VM PROD       |                                                                                      .------------------------.
 | production services |                                                                                      |       other VMs        |
 |---------------------|                                                                                      |------------------------|
 | Ubuntu 18.04 on LVM |<----------------------------------------------------------------------------------.  | for testing            |
 | with LXD containers |   |                          |                          |                         |  | and learning purposes  |
 | on ZFS              |   |                          |                          |                         |  | (CentOS, pfsense etc)  |
 | +br0 bridge         |   |                          |                          |                         |  '------------------------'
 '---------------------'   |                          |                          |                         |
            ^              |                          |                          |                         |
            |              |                          |                          |                         |
            |              |                          |                          |                         |
            |  .----------------------. .--------------------------. .-----------------------. .-----------------.
            |  |   LXC container 2    | |     LXC container 3      | |    LXC container 4    | | LXC container 5 |
            |  |----------------------| |--------------------------| |-----------------------| |-----------------|
            |  | nextcloud            | | web applications:        | | XMPP server (prosody) | | SAMBA           |
            |  | (cloud.mydomain.com) | | gitea, wallabag etc      | | (xmpp.mydomain.com)   | | (LAN only)      |
            |  '----------------------' | (git.mydomain.com, etc.) | '-----------------------' '-----------------'
            |              ^            '--------------------------'             ^                      ^
            |              *                                 ^                   *                      |
 .---------------------.   *                                 *                   *                      |      Home PC, 
 |   LXC container 1   |   *                                 *                   *                     L|     laptop etc
 |---------------------|**********************************************************                     A|----- __  _    
 | nginx reverse proxy |                                                                               N|     [__]|=|   
 '---------------------'                                                                                |     /::/|_|   
            ^                                                                                           |
            *                                                                                           |               .-,(  ),-.    
            *                                            access from INTERNET                     Home router        .-(          )-. 
            ************************************************************************************** __________ <*****(    internet    )
                                                                                                  [_...__...°]       '-(          ).-'
                                                                                                                         '-.( ).-'    

QUESTIONS

  1. How to orchestrate / administer it? Since every container / VM is a full separate OS, how do I batch upgrade them? With Ansible?
  2. HP Proliant has two NICs - how can I isolate home LAN (samba) with internet?
  3. Should I put samba on HOST rather than in LXD for performance reasons?
  4. How do I manage storage? To expose folders/ disk to containers I have to share it with VM PROD and then with LXD - is there a big penalty in I/O for it? Can I go with ZFS (since HOST is Ubuntu 18.04 which supports it without DKMS)
  5. Is there a big performance fall in setting ZFS storage for LXD containers on LVM?

Any additional comments and shared experience would be much appreciated.


#2

Hi mDfRg,

Welcome to our forums.

Quite a plan you got there :slight_smile: Makes me feel I should add some more hardware to my home lab.

About your questions:

  1. Batch upgrade: I’d advise upgrading the containers first (preferably the ones with the same OS in the test/dev environment, and after verification prod containers too), then the VMs, then bare metal HOST the same way. If network bandwidth may be an issue, put some delay between them/implement sequential upgrade. Orchestrate/administer: by “how” what do you mean?

  2. Just to clarify: you want to setup 1 NIC for LAN-only and 1 NIC for Internet-facing services?

  3. and

  4. For every layer of virtualization, there is always a tradeoff in performance (but there is also the plus side of separation). Since you planned the HOST to be as minimal as possible, I’d say Samba should be in a container too - especially since it can have issues from time to time. On the other hand, it depends on use case, you may not even notice the difference in I/O with and without containers. You can use ZFS, LXD should support it without problem.

  5. The fact that containers are on LVM and the storage provided for them is ZFS should not come with any overhead. The virtualization layers do put some on performance, but that’s mostly memory and processing power, as disks are local in any case in your setup (not mentioning mounting the Samba share via network).

You haven’t mentioned how the port forwarding is set up on the router, but in general I advise to move the ports forwarded from defaults, as an open port 80 on any public IP address is a red sign for bad actors there is a webserver for breaking, and an open port 22 says “hey, try a few million username/password on me”. Of course this advice only applies if there are no public services on the server, and it is only for personal use.