Updating grub and reboot to load the new kernel parameters to isolate specific cores:
sudo update-grub
Makes the host don’t schedule any job on those cores by looking at the workload per core using:
htop
Starting a LXD vm pinning it on those cores does not seems to be scheduling jobs on them anymore, as it used before the latest lxd release/kernel update.
lxc launch ubuntu:24.04 test --vm -c limits.cpu=3,8
lxc shell test
root@test:~# for i in $(seq $(getconf _NPROCESSORS_ONLN)); do yes > /dev/null & done
# Output the following
#[1] 1232
#[2] 1233
#[3] 1234
#[4] 1235
htop on the vm shows all 2 cores at 100%.
htop on the host show cores (other than 3 or 8) spiking at 100%, it seems like lxd vm is now scheduling the workload on the cores available to the host instead of the isolated one reserved exclusively for the lxd vm.
This used to be working fine earlier this year 2024. Any ideas how I can make the lxd queue vm workload on isolated cores?
Experienced it on 6.1-78a3d8f.
Then rolled back to latest version 5.21 (5.21.2-2f4ba6b) and it seems like its also showing the same issue. In the meanwhile, I removed isolcpu parameter from kernel 5.15.0-119-generic running Ubuntu 22.04.
Let me try it on a non production server to see if lxd 6.2 is able to schedule vm/container workloads on host unpinned cpu cores and I will report back my findings.
The issue here is that for containers we explicitly forbid using isolated CPUs even with explicit cpu pinning (see Add support for isolcpu in CPU scheduler · canonical/lxd@257bc06 · GitHub). But after we’ve added auto cpu pinning feature we started to use a uniform approach to determine CPU pins for containers & VMs which changed behavior for VMs and even explicit pinning of isolated cores became impossible. This is obviously wrong and we should fix it.