Microceph on LXD

Project Discussion Ceph
1

Hello! Total newbie here, so sorry for the dumb question.

For a start, I have a small LXD cluster (two hosts) using the Ubuntu Fan network, and I’d like to have Ceph available for some development. Specifically, I have a BOSH LXD CPI that enables BOSH to be deployed into LXD. I have cluster operations working, but all the disk management logic uses local storage – so the CPI has logic to copy volumes between hosts should a VM get repositioned.

I would like to setup Ceph and use that as a basis for remote store pools in LXD. However, the tutorials in the documentation really don’t discuss running in a VM. I did find what someone else did to configure Microceph in 3 VMs.

It works… For a bit. And then hosts start dropping out. With 3 VMs, 2 are on the same host and the 3rd is on the other host. First, that 3rd one drops out and then (sometime over night) the other two left.

All of this is on current snaps of LXD 5.21 LTS, and whatever is current for Microceph.

So the general questions are:

  1. Can Microceph/ceph be run in a VM (not looking for most performant, just stability).
  2. Are there requirements on the VM (CPUs and/or memory?).
  3. Is there anything around networking I should worry about? tracepath seems to indicate the hosts actually direct network traffice between the hosts and not back to the router, so I suspect this is mostly ok.

Thanks!

2

Here’s what the health looks like:

root@microceph-1:~# microceph.ceph -s
  cluster:
    id:     45ddfd0e-d7c8-4a24-b42b-4a83281b2bb2
    health: HEALTH_WARN
            2 osds down
            3 hosts (3 osds) down
            1 root (3 osds) down
            Reduced data availability: 65 pgs inactive
            1 pool(s) do not have an application enabled
 
  services:
    mon: 3 daemons, quorum microceph-1,microceph-2,microceph-3 (age 12h)
    mgr: microceph-2(active, since 16h), standbys: microceph-3, microceph-1
    osd: 3 osds: 0 up (since 11h), 2 in (since 20h)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             65 unknown
3

Hey! I’d be interested in knowing what exactly is your resilience requirements ? Do you need three replicas ? If you are not distributed across 3 hosts that may not even be helpful (etc).

4

Hey @utkarshbhatthere! I forgot I had posted this question. :slight_smile:

I wasn’t looking for anything specific – just that I had an LXD cluster and wanted to use remote storage for VMs in that cluster. But I was trying to do a proof-of-concept before I went and purchased more hardware. I believe that it was more stable at 4 VMs with an increase in memory. (This was a few months ago and I don’t remember the specific details.) At any rate, it was stable enough that I purchased a 3rd host.

My current setup is 3 used Xeon workstations with 64GB and pretty much every SSD had sitting around. Microceph seems to be pretty stable and content with that mishmash of disks. I think it has survived one hard-disk failure. And it was doing it’s best when a memory DIMM went bad (I didn’t realize they actually do go bad sometimes).

Regardless, so far so good! It’s more complex to run than LXD, but the microceph tooling helps a bit. I’ve learned that when I “do maintenance” on the hosts, I shutdown all VMs (being a home lab allows this!) and then put microceph into maintenance mode. This allows me to update and reboot at will. Seems to work rather well.

Thanks!

1 Like
5

Awesomesauce, feel free to chime in our Matrix rooms for instant-ish chats with the team here. Feedback of homelab operators is most welcome.

1 Like