Microceph on LXD

Project Discussion Ceph
1

Hello! Total newbie here, so sorry for the dumb question.

For a start, I have a small LXD cluster (two hosts) using the Ubuntu Fan network, and I’d like to have Ceph available for some development. Specifically, I have a BOSH LXD CPI that enables BOSH to be deployed into LXD. I have cluster operations working, but all the disk management logic uses local storage – so the CPI has logic to copy volumes between hosts should a VM get repositioned.

I would like to setup Ceph and use that as a basis for remote store pools in LXD. However, the tutorials in the documentation really don’t discuss running in a VM. I did find what someone else did to configure Microceph in 3 VMs.

It works… For a bit. And then hosts start dropping out. With 3 VMs, 2 are on the same host and the 3rd is on the other host. First, that 3rd one drops out and then (sometime over night) the other two left.

All of this is on current snaps of LXD 5.21 LTS, and whatever is current for Microceph.

So the general questions are:

  1. Can Microceph/ceph be run in a VM (not looking for most performant, just stability).
  2. Are there requirements on the VM (CPUs and/or memory?).
  3. Is there anything around networking I should worry about? tracepath seems to indicate the hosts actually direct network traffice between the hosts and not back to the router, so I suspect this is mostly ok.

Thanks!

2

Here’s what the health looks like:

root@microceph-1:~# microceph.ceph -s
  cluster:
    id:     45ddfd0e-d7c8-4a24-b42b-4a83281b2bb2
    health: HEALTH_WARN
            2 osds down
            3 hosts (3 osds) down
            1 root (3 osds) down
            Reduced data availability: 65 pgs inactive
            1 pool(s) do not have an application enabled
 
  services:
    mon: 3 daemons, quorum microceph-1,microceph-2,microceph-3 (age 12h)
    mgr: microceph-2(active, since 16h), standbys: microceph-3, microceph-1
    osd: 3 osds: 0 up (since 11h), 2 in (since 20h)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             65 unknown