An update on the licence change and community image server

sekurilab · January 26, 2024, 8:43am

Don’t prioritize, just do what they are doing. Build the images with distrobox, setup CI/CD and host the images. This way we will get all the images we had before. This change is already affecting users in production environments which is really bad. Waiting for May will make things even worse and a lot of people will migrate to Incus because of the delay and LXD will lose users. It’s not like we were not warned about it by linux containers maintainers.

zacksiri · January 30, 2024, 5:50am

Any progress update on this? We’d love to know how it’s progressing.

holmanb · January 31, 2024, 6:11pm

Don’t prioritize, just do what they are doing.

I’d like to echo @sekurilab’s suggested course of action. Cloud-init needs to be able to test everything we support. See our supported distro list here if you need to prioritize. In the meantime we can use Incus locally for manual testing, but the tooling we use for testing currently only supports lxd - regaining lost testing capabilities ASAP would be best.

knutov · February 3, 2024, 11:37am

Please add Centos 7.

btw, is there a way to build self-hosted image server and documentation how to build images?

Lestat · February 4, 2024, 1:59am

i have a controversial request, for which incus are weak as well - create a way to :

create own self-hosted repositories via http/simplestreams
instruction how to transform supported OS Distro ISO to LXD/LXC Image
how to integrate it to the hosted repository

Probably some integration to the Sonatype Nexus OSS could be made, or own docker/lxc based image with the “image” server for lxd & simplestreams/http proto.

P.S.
And yes, coz of delay i already forced to migrate my home-lab to incus coz i’m fedora server user alongside of the ubuntu and currently stomped on how bad situation is with creating own images and hosting them are in Incus (linuxcontainers) and LXD (canonical)

Lestat · February 4, 2024, 2:17am

he’s not one of the maintainer but core developer actually, and what he described is basically from which point Incus forked from LXD but in the future, nobody guarantee their compatibility as well on how bright would be Incus future(or how much that guy would be able to support the infra?).

On LXD side nothing is better - think is that for half of year nothing were made to create/migrate to own LXD repositories, alongside the “ubuntu:” one and absence of possibility and clear guides on how to create own ones raises a concern, especially for environments without internet.

zacksiri · February 12, 2024, 9:19am

Hi All

I’ve begun working on the image server, and also setup the build system for the images. You can find more info here. The goal is to get something up and running within 2 weeks.

Image Server - https://github.com/upmaru/polar
Builder - https://github.com/upmaru/opsmaru-images

We’re prioritizing getting and MVP up and running, given that we have limited resources I’m going to be focusing primarily on alpine container images for aarch64 and amd64 for lxd. That’s the basic setup we need for our product.

The basic build configuration is there but it’s still missing one piece that pushes the built image to the server which I will solve in due time.

I’m making good progress and am positive about getting the MVP up in 2 weeks. If there is any particular OS you’d like to see, feel free to submit an issue on the opsmaru-images or submit a pull request. Documentation is a little sparse right now but you can read the code it’s pretty straight forward.

Project Goals

To create an open image server that is transparent and can be easily managed by the community.

Deployment Options

Users will either be able to sign up to our image server, get a token and use the image server managed by upmaru OR users will be able to host their own image server.

MVP Features

Provide basic functionality for serving lxd / incus images
Provide a way to build / push images to S3 compatible storage
Provide a way to use CDN in-front to ensure fast performance

Roadmap

Some UI to enable easy management for users to issue their own tokens.
Some UI to enable users to Bring their own CDN so they can be responsible for their own bandwidth / usage.

zacksiri · February 15, 2024, 11:48am

I wanted to update everyone on my progress.

I got the hash calculation working, and have setup the repo that will host the github action.

You can see how the metadata for the items are being built here:

https://github.com/upmaru/icepak/blob/develop/lib/icepak/push.ex

[
  %{
    name: "incus.tar.xz",
    size: 880,
    path: "images/alpine/3.19/amd64/default/1/incus.tar.xz",
    source: "/home/zacksiri/test/incus.tar.xz",
    file_type: "incus.tar.xz",
    hash: "45c9f70358879bc8163843ef86d51592a6ff8d1177db0f1df9b6c6e571322eab",
    combined_hashes: [
      %{
        name: "combined_squashfs_sha256",
        hash: "0d0c64a632ead7ac29516562cdcff63359e49544214ac08dd2dbc43aa9fc3ed4"
      }
    ],
    is_metadata: true
  },
  %{
    name: "lxd.tar.xz",
    size: 880,
    path: "images/alpine/3.19/amd64/default/1/incus.tar.xz",
    file_type: "lxd.tar.xz",
    hash: "45c9f70358879bc8163843ef86d51592a6ff8d1177db0f1df9b6c6e571322eab",
    combined_hashes: [
      %{
        name: "combined_squashfs_sha256",
        hash: "0d0c64a632ead7ac29516562cdcff63359e49544214ac08dd2dbc43aa9fc3ed4"
      }
    ],
    is_metadata: true
  },
  %{
    name: "root.squashfs",
    size: 2850816,
    path: "images/alpine/3.19/amd64/default/1/rootfs.squashfs",
    source: "/home/zacksiri/test/rootfs.squashfs",
    file_type: "squashfs",
    hash: "b5409f8817f9e297c51fa96a426f5876c20447c1b1767902ea84f199853889b9"
  }
]

The Goal of the icepak repo is it will run inside the github action. It will essentially do the following:

Push built artifacts to S3 storage
Compute all the hash / metadata necessary
Create version for a particular product

This will essentially give polar all the information needed to generate:

/streams/v1/index.json
/streams/v1/images.json

Those two json endpoints are basically based on simplestream protocol.

gabriel-samfira · February 16, 2024, 9:40am

I don’t mean to bug anyone, but is there an update on this. We had to move to Incus on one project due to the fact we needed a Debian image. Both projects are great, but I would have loved to not spend a day migrating things.

Also, it would be great to have Ubuntu images with generic kernel instead of the “kvm” kernel for VMs, by default.

tomp · February 16, 2024, 10:14am

We are actively working on it.

WRT to generic kernels, I believe from 23.10 onwards (including the forthcoming 24.04 LTS release) the images from ubuntu: and ubuntu-daily: are using the generic kernels.

lxc launch ubuntu:23.10 --vm v1
Creating v1
Starting v1                                   
lxc exec v1 -- uname -a
Linux v1 6.5.0-17-generic #17-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 11 14:01:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

gabriel-samfira · February 16, 2024, 12:03pm

That’s perfect! The kvm one is missing LVM2 and binfmt (needed for cross compilation) support.

Thanks for the update!

zacksiri · February 19, 2024, 11:57am

Hi All

So I managed to get the whole flow working locally, this means the following:

Icepak (github action) is able to push built images to polar.
Polar is able to generate LXD specific feed OR Incus specific feed.
Polar is able to serve images correctly using CDN.
Users will be able to BYOCDN (bring your own CDN) if they so choose.

I made a high-level architecture diagram to make things easier to understand.

zacksiri · February 21, 2024, 12:19pm

Hi All

Another update on this. So the MVP (sandbox) server is up and running if you would like to try it out it’s here:

https://images.opsmaru.dev

The production server:

https://images.opsmaru.com

How to use

To use the image server you can use this url (expires in 30 days)

lxc remote add opsmaru https://images.opsmaru.dev/spaces/b82cb0551bccd5edb7cf8626 --public --protocol simplestreams

What happens after this expires?

We are working on a UI where you can sign up and issue your own tokens / url. You will also be able to choose between LXD / incus feed. The image listing will be customized based on your choice. This is to ensure the feed is clean and doesn’t cause problems with either of the clients. The example above is and LXD feed.

You will be able to issue as many tokens as you want including tokens that never expire, you should keep your token private and not share it with anyone.

You don’t have the images I need

We will add new images, if there is demand, please open an issue or a pull-request here.

We will currently support the following architectures:

x86_64 (amd64)
aarch64 (arm64)

Will consider adding more if there is demand.

The bulk of the build system is done, I’m using a fork here which pushes to the sandbox server.

You can see the success build action here.

When will the ui be done

This is something of a high priority for us, therefore soon. I’m working on cleaning up the MVP ui to get this thing up in production hopefully beginning March.

Can I self host the image server?

Yes you will be able to self-host the image server if you want. We will provide instructions and an easy guide to enable you to do this.

In action

lxc image list opsmaru:

+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
|           ALIAS            | FINGERPRINT  | PUBLIC |           DESCRIPTION           | ARCHITECTURE |   TYPE    |  SIZE   |          UPLOAD DATE          |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.16 (3 more)       | d4e280b3b850 | yes    | alpine 3.16 arm64 (20240221-14) | aarch64      | CONTAINER | 2.28MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.16/amd64 (1 more) | 4fbbab01353e | yes    | alpine 3.16 amd64 (20240221-14) | x86_64       | CONTAINER | 2.50MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.17 (3 more)       | 8edf37df13ec | yes    | alpine 3.17 arm64 (20240221-14) | aarch64      | CONTAINER | 2.70MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.17/amd64 (1 more) | 099f83764a67 | yes    | alpine 3.17 amd64 (20240221-14) | x86_64       | CONTAINER | 2.93MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.18 (3 more)       | 7c31777227b0 | yes    | alpine 3.18 arm64 (20240221-14) | aarch64      | CONTAINER | 2.75MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.18/amd64 (1 more) | 37062029ee44 | yes    | alpine 3.18 amd64 (20240221-14) | x86_64       | CONTAINER | 2.94MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.19 (3 more)       | e44e496455f5 | yes    | alpine 3.19 arm64 (20240221-14) | aarch64      | CONTAINER | 2.72MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/3.19/amd64 (1 more) | b392f4461aaf | yes    | alpine 3.19 amd64 (20240221-14) | x86_64       | CONTAINER | 2.92MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/edge (3 more)       | 34b71a8b87ab | yes    | alpine edge arm64 (20240221-14) | aarch64      | CONTAINER | 2.72MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+
| alpine/edge/amd64 (1 more) | 4d7c0a086c41 | yes    | alpine edge amd64 (20240221-14) | x86_64       | CONTAINER | 2.93MiB | Feb 21, 2024 at 12:00am (UTC) |
+----------------------------+--------------+--------+---------------------------------+--------------+-----------+---------+-------------------------------+

zacksiri · February 26, 2024, 10:53am

We have recently added debian and ubuntu images to the image server. You can try it out.

debian/bookworm
debian/sid
ubuntu/jammy

Also in case anyone is curious about our CDN we are using bunny.net you can see their network coverage here:

igormahov · February 26, 2024, 6:31pm

Do these images contain cloud-init? I used it to initialize VMs when used images linuxcontainers

zacksiri · February 26, 2024, 9:18pm

You mean the cloud variant? We can add that. You’ll also need the VM images right? We can add that too. I’ll add them and keep you posted once they’re up.

zacksiri · February 27, 2024, 4:29pm

Hi Everyone

Another update from me. I just shipped an updated version of the sandbox image server

https://images.opsmaru.dev

You should be able to sign up and issue your own tokens that don’t expire now. Do note that the production server is also coming up and will have monitoring / redundancy.

sekurilab · February 29, 2024, 8:47am

This is very cool to provide the source for the web and image workflows. I’ll self-host and when I find time contribute to the image repo. Thank you for your effort, it’s much appreciated.

zacksiri · March 5, 2024, 2:58am

Hi Everyone

The production image server is up and running here:

https://images.opsmaru.com

We’ve switched our internal infrastructure to using this server. The difference between the production and sandbox is that, production has redundancy and health check. You can see the status here https://status.opsmaru.com

Production will also not be updated as regularly as the sandbox version. Only well tested changes will make it to production. This is to ensure maximum stability given that the image server as simple as it maybe is a core infrastructure component.

tomp · April 4, 2024, 8:18am

We have now launched our new non-Ubuntu image server: