Chisel manifest is supported in newly released v1.0.0

Rocks Announcements
1

Chisel v1.0.0 has been released recently! It comes with a bunch of cool features. This post describes the new features along with a basic explanation. (Jump to the new stuff.)

Basics

What is Chisel again?

Chisel can derive a minimal Ubuntu-like Linux distribution using a release database that defines “slices” of existing packages. Slices enable developers to cherry-pick just the files they need from the Ubuntu archives, and combine them to create a new filesystem which can be packaged into an OCI-compliant container image or similar.

In summary, with Chisel, you get to choose which files you want to have in your rootfs and extract only those from the packages.

That sounds cool! How do I use it?

Using Chisel is super easy. First, install it the way you want –

  • Compile with go:

    go install github.com/canonical/chisel/cmd/chisel@latest

  • Install the snap:

    snap install chisel --candidate

  • Download the pre-built binaries from the release
    page    :

    # Download the Chisel tarball.
wget https://github.com/canonical/chisel/releases/download/v1.0.0/chisel_v1.0.0_linux_amd64.tar.gz
# Download the hash and validate.
wget https://github.com/canonical/chisel/releases/download/v1.0.0/chisel_v1.0.0_linux_amd64.tar.gz.sha384
sha384sum -c chisel_v1.0.0_linux_amd64.tar.gz.sha384
# Install the Chisel binary.
tar -C /usr/bin -xf chisel_v1.0.0_linux_amd64.tar.gz

Check your installation with the following:

$ chisel version
v1.0.0

Finally, you can install your slice(s) like the following:

mkdir rootfs
chisel cut --release ubuntu-24.04 --arch amd64 --root rootfs/ hello_bins

This should install the hello binaries in your rootfs. You can check if it works with the following:

$ chroot rootfs/ hello
Hello, world!

What is a slice?

A Debian/Ubuntu package is a collection of different types of files – binaries, libraries, configuration files, copyright, docs etc. The philosophy of Chisel is that you may not need all the files from each package.

Chisel introduces a slice as a (partial) group of package files. Thus, a package can have many slices and they can overlap as well! In fact, you can have an unlimited amount of slices per package, Chisel won’t complain.

Well, where are these slices then?

Chisel uses slice definition files to describe the slices for a package. The collection of these files with a configuration file named chisel.yaml is called a chisel release. The chisel-releases repository contains a number of releases for various Ubuntu versions.

To use any of these releases, pass the branch name as the argument to the --release option in the cut command. For example, to use the Focal (20.04) release, you would do something like this:

chisel cut --release ubuntu-20.04 --root rootfs/ hello_bins

The --root option is required and is used to specify a directory where chisel should install the files.

How do I know which slices are there?

Check out the chisel-releases repository! The structure of the files are as follows:

chisel.yaml
slices/pkg-name.yaml
slices/...

The slices for a particular package foo typically can be found at slices/foo.yaml.

Or, use the new find command!

Can I have my own slices?

Sure! The --release option accepts a path to your custom release directory. You can edit, add, remove slices in your local release directory as you want. See the README.

We would love it if you kindly contribute your slices to the upstream chisel-releases repository! See CONTRIBUTING.

Commands

Chisel commands can be classified as follows:

  • Basic (general operations):

    • find: Find existing slices
    • info: Show information about package slices
    • help: Show help about a command
    • version: Show version details

  • Action (make things happen):

    • cut: Cut a tree with selected slices

The cut command

The cut command is the only action command which installs the selected slices along with their dependencies, for a particular chisel-release (specified by --release, optional) and a particular package architecture (specified by --arch, optional) in a particular directory (specified by --root).

Usage:
  chisel cut [cut-OPTIONS] [<slice names>...]

If the --release and/or --arch options are not provided, they are inferred from the host system Chisel is runing on.

The find command

Chisel v0.10.0 introduced the find command which can be used to reduce the hassle of going to each branch and checking if there exists slices for a package.

$ chisel find hello_* 2>logs
Slice            Summary
hello_bins       -
hello_copyright  -

Read more about the find command in this discourse post.

The info command (NEW)

The info command complements the find command. Whereas the find command searches for existing slices, the info command takes in slice or package name(s) and displays the slice definition(s).

Here’s an example:

$ chisel info hello_bins 2>logs
package: hello
archive: ubuntu
slices:
    bins:
        essential:
            - hello_copyright
            - libc6_libs
        contents:
            /usr/bin/hello: {}

Unlike the find command, the info command does not accept wildcards. But it does take package names as query strings and it can process multiple queries at once. Different YAML documents are separated by three dashes ---.

$ chisel info hello libgcc-s1_libs 2>logs
package: hello
archive: ubuntu
slices:
    bins:
        essential:
            - hello_copyright
            - libc6_libs
        contents:
            /usr/bin/hello: {}
    copyright:
        contents:
            /usr/share/doc/hello/copyright: {}
---
package: libgcc-s1
archive: ubuntu
slices:
    libs:
        essential:
            - libgcc-s1_copyright
            - libc6_libs
        contents:
            /usr/lib/*-linux-*/libgcc_s.so.*: {}

It takes only one option, --release which is similar to the find and cut commands. For more info, see chisel help info.

Chisel manifest (NEW)

This is a much anticipated feature of Chisel which has been in the works througout different releases. Chisel manifest is a ZSTD-compressed file which lists the metadata about installed packages, slices and most importantly, files.

The generate keyword

v1.0.0 also introduced a new generate keyword for file paths in slice definitions. The generate keyword currently accepts only one non-empty value: manifest. And the generate keyword can be specified on certain type of glob paths in the following format:

/path/to/dir/**: {generate: manifest}

The path must be a directory and must end with double-asterisks (**) which matches everything inside that directory. Additionally the path must not contain any other wildcard characters (*, ?) e.g. the /path/to/dir part must not have any wildcard. These are treated as typical globs while checking for conflict and these directories and everything inside those are generated by Chisel.

If a slice with a generate: manifest-specified path is selected to be installed, Chisel writes the manifest to a manifest.wall file inside the specified directory. In the above example, the manifest path would /path/to/dir/manifest.wall. If there are multiple generate: manifest-specified paths, the manifest is generated at each of those places.

The base-files_chisel slice

By default, chisel-release provides a base-files_chisel slice for convenience in each of its releases, which contains a generate: manifest-specified path.

package: base-files
essential:
  - base-files_copyright
slices:
  # Dedicated slice for generating the Chisel manifest.
  chisel:
    essential:
      - base-files_var
    contents:
      /var/lib/chisel/**: {generate: manifest}
  ...

Thus, if you install the base-files_chisel along with other slices, a manifest file would be generated at /var/lib/chisel/manifest.wall. This file is ZSTD-compressed.

$ chisel cut --root rootfs/ hello_bins base-files_chisel
...
...
2024/10/17 11:49:07 Generating manifest at /var/lib/chisel/manifest.wall...

Manifest format

The uncompressed file is in jsonwall-format – a collection of JSON objects, one-per-line. The lines except the header are sorted in lexicographical-order for better compression.

jsonwall Header

The header is a single JSON object on the first line in the following format:

{"jsonwall":"1.0","schema":"1.0","count":84}

This JSON object has the following attributes:

Field Type Required Description
jsonwall str required the version of jsonwall.
schema str required the schema version that Chisel manifest uses.
count int required the number of JSON entries (or, lines) in this file, including the header itself.

Packages

For each package installed in the file system, a JSON object with "kind":"package" will be present in the manifest.

{"kind":"package","name":"foo","version":"1.0-2","sha256":"abcd...","arch":"amd64"}

These JSON objects have the following attributes:

Field Type Required Description
kind str required type of JSON object – the value will always be package.
name str required name of the package.
version str required version of the package.
sha256 str required digest of the package (in hex format).
arch str required architecture of the package.

Slices

For each slice installed in the file system, a JSON object with "kind":"slice" will be present in the manifest.

{"kind":"slice","name":"foo_bar"}

These JSON objects have two attributes:

Field Type Required Description
kind str required type of JSON object – the value will always be slice.
name str required name of the slice, in the pkg_slice format.

Paths

For each path defined in the slice definitions that Chisel installs in the file system, a JSON object with "kind":"path" will be present in the manifest.

{"kind":"path","path":"/etc/","mode":"0755","slices":["foo_bar","abc_xyz"]}
{"kind":"path","path":"/etc/foo","mode":"0644","sha256":"abcd...","final_sha256":"abcd...","size":1234,"slices":["foo_bar"]}
{"kind":"path","path":"/etc/bar","mode":"0777","link":"/etc/foo","slices":["foo_bar"]}

These JSON objects have the following attributes:

Field Type Required Description
kind str required type of JSON object – the value will always be path.
path str required location of the path.
mode str required the permissions of the path, in an octal value format.
slices list required the slices that have added this path.
sha256 str optional the original checksum of the file as in the package (in hex format). This attribute is required for all regular files, except the manifest.wall file itself, which is an exception.
final_sha256 str optional the checksum of the file after it has been modified during installation (in hex format). This attribute is required only for files that have been mutated.
size int optional the final size of the file, in bytes. This attribute is required for regular files, except the manifest.wall file itself, which is an exception.
link str optional the target, if the file is a symbolic link.

Note: unless explicitly included in the requested slices, the parent directories that are implicitly created in the filesystem will not be written to the Chisel manifest. In the example above, it is assumed that /etc/ is explicitly defined, with the same properties, in both foo_bar and abc_xyz.

List of Paths under a Slice

To state the path that a slice has added/changed, JSON objects with "kind":"content" are used.

{"kind":"content","slice":"foo_bar","path":"/etc/foo"}

It has the following attributes:

Field Type Required Description
kind str required type of JSON object – the value will always be content.
slice str required name of the slice.
path str required location of the path.

New features in v1.0.0

That’s all for now. :slight_smile: Please let me know in comments if you have any questions.

3 Likes