Managing or developing for a multi-cloud deployment might sometimes require an engineer to collect the most up-to-date images for one or more clouds, or to fetch specific images. The engineer could write separate tools for each cloud, and even hide them behind a common abstraction; however, if the engineer is using Ubuntu images, we’ve already created a solution for them, which we call Simple Streams.
Simple Streams are a sub-service of cloud-images.ubuntu.com; they are metadata streams together with a suite of tools for parsing and filtering them. In addition to downloadable Ubuntu cloud images for each supported release and architecture, cloud-images.ubuntu.com publishes metadata in the form of JSON that provides information describing Ubuntu images on major public cloud platforms. These metadata streams are signed and updated with every new Ubuntu daily or release publication.
Canonical has created a couple of tools to help parse and display these metadata streams.
The simplest is called “image-status.” It provides a basic interface for querying the very latest images on the Simple Streams feeds, and is available in the form of a Snap package. This script provides an example of how image-status might be used to gather the most up-to-date image serials across the AWS EC2, GCE and Azure clouds.
Here are some usage examples:
Get the latest recent-LTS (bionic) daily serial for each cloud:
$ ./latest-images.sh
Getting latest bionic ec2-daily image...
AWS: 20190905
Getting latest bionic gce-daily image...
GCE: 20190904
Getting latest bionic azure-daily image...
Azure: 20190903
Get the latest disco minimal release serial for just AWS EC2:
$ ./latest-images.sh -c ec2 -r disco -t release -m
Getting latest disco ec2-release-minimal image...
AWS: 20190903
Another tool, which provides a more granular interface to querying Simple Streams metadata, is called ‘sstream-query’, and is shipped along with the Simple Streams code. Canonical has also created a snap of this tool to make it easier to install and use. Sstream-query is a more direct interface to Simple Streams than image-status, and allows the user to look for more than just the most recent images. Combining the output of sstream-query with a tool like ‘jq’ enables powerful searching and filtering. Here is an example script, similar to the one above, that allows a user to gather recent or historical image serials based on a date expression.
Usage examples:
Get the current month’s most recent bionic daily serial for each cloud (the script queries the current month if no date is given):
$ ./query-images.sh -r bionic -t daily
Getting latest Bionic daily serials from date 201909...
AWS:
20190905
GCE:
20190904
Azure:
20190903
Get the previous month’s last bionic daily serial for just GCE (the -p switch tells the script to query whatever month is prior to the current month):
$ ./query-images.sh -r bionic -t daily -p -c gce
Getting latest bionic daily serials from date 201908...
GCE:
20190828
Get all the Xenial release images from June 2019 by passing a YYYYMM date:
$ ./query-images.sh -r xenial -t release -n 30 -d 201906
Getting latest xenial releases serials from date 201906...
AWS:
20190628
20190617
20190605
GCE:
20190628
20190617
20190605
Azure:
20190628
20190625.2
20190617
20190605
Simple Streams metadata also provides cloud-specific information, such as image-id. Passing the -i switch to query-images.sh will emit the cloud-specific image id for each serial:
$ ./query-images.sh -t release -p -i
Getting latest bionic releases serials from date 201908...
AWS:
20190814: ami-09a3d8a7177216dcf
GCE:
20190813a: ubuntu-1804-bionic-v20190813a
Azure:
20190813.1: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-18_04-LTS-amd64-server-20190813.1-en-us-30GB
These scripts do make a couple of shorthand assumptions about the type and location of the image being looked for: e.g. they query the United States pacific northwest region of each cloud, and in the case of AWS EC2 look for images with the latest root storage and virtualization types, which are supported for Precise (12.04) and newer, but may not exist prior to that. Further modifications of these scripts could expose those axes to their option sets.
It is also worth noting that, depending on the Ubuntu release being queried (e.g. ‘precise’ or ‘bionic’), images may not exist in all clouds.
As long as you’re using Ubuntu, though, Simple Streams provides you with a one-stop-shop for all your metadata needs.
By the way, further discussion of sstream-query and image-status has been detailed in a series of excellent blog posts by Canonical’s very own Phil Roche, who is a member of the Certified Public Cloud team.