Checkbox 7.1.0 stable release

Project Discussion Checkbox
1

Hello all!

A new version of Checkbox is now available as a Debian package (in the Stable PPA ) and as Snaps (in the stable channel)!

In this release we have added two much anticipated features to Checkbox.

First, Checkbox now supports reading units in YAML! This release introduces a new parser and a translator to automatically allow you to migrate over, along with YAML linting support (pre-configured in the monorepo for VS Code).

Secondly, we have started collecting much richer system metadata directly into your JSON submissions without relying on mandatory inclusions. This should result in submission (more reliably) containing all the important metadata to have a good picture of the system that run the tests (Ubuntu and kernel versions, BIOS and memory information, etc.).

This release also bundles new tests for NPUs, modern connectivity standards like Wi-Fi MLO and Bluetooth 6.0, graphics and compute with expanded Vulkan, OpenCL, and CUDA coverage, along with the usual huge pack of bugfixes.

Reminder: if you update Checkbox you will not be able to resume testing session that you begun before updating. If this is undesirable to you, consider pinning Checkbox and finishing all sessions that are ongoing with the following command:

# for snaps:
$ sudo snap refresh --hold=forever checkbox checkbox22
# for debs:
$ sudo apt-mark hold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

Remember to eventually run the following to re-enable automatic updates:

# for snaps
$ sudo snap refresh --unhold checkbox checkbox22
# for debs
$ sudo apt-mark unhold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

New in this release

  • Add vulkan cts to base provider
  • Add fscrypt tests to client-cert-iot-24-04 test plan
  • Graphics_card: add drm_node field
  • Add kernel modules to submission json
  • Use checkbox-pipewire-utils default_device_is_real in test jobs
  • Opencl cts classic test plan
  • Add custom-frontend slot to checkbox-ce-oem
  • Yaml parser
  • Remove qr tests
  • Add bios mode to submission json
  • Refactor meta package test case in pc-sanity
  • Add opencl-cts to base provider
  • Add bios metadata to submission json
  • Add remoteproc fw check
  • Include check-hardware-attributes to pc-sanity
  • Check attributes under /sys can be read without stucking the system
  • Add snap packages to metadata collected in json report
  • Migrate rz camera test for koto
  • Detect dummy/virtual sinks with checkbox-support-pipewire-utils
  • Add device information to any submission json generated by checkbox
  • Add certification test plans for ubuntu 26
  • Update docking certification test plans
  • Add credit card reader test
  • Npu test plan for known failures
  • Add npu tests
  • Translator from pxu to yaml
  • Cpuid.py fixes and supported intel platforms update
  • Add screen-private-ppa test
  • Support extra envvars per snap
  • Add test-snapd-tools-core24 support
  • Add smartcard test
  • Add cash drawer test
  • Add printer test
  • Add webgl conformance test
  • Test rpmsg function by checking each subsystem and device tree
  • Add barcode scanner test
  • Providers/base: add fscrypt test
  • Migrate to new randr cycle script
  • Audio server utils module
  • Update gl support test to use glmark2
  • Add codec test for koto
  • Migrate pc-sanity to new script for gpu drivers
  • Add imx8m video codec test
  • Add level zero raytracing testing
  • Extend camera tests for nxp imx8mp evk
  • Add pwm fan tests
  • Add the nvidia-requested vk cts subset
  • Add screen-private-ppa test
  • Systemd-unit based runner
  • Add serial stress test
  • Add a test case to verify the test audio i/o after suspend with previously occupied devices
  • Add bluetooth 6.0’s hci version in beacontools
  • Refactor bios error test script with python and add unit tests
  • Add a simple wifi mlo test
  • Custom frontend interface
  • Add level zero testing
  • Add running dss integration tests
  • Add checkbox-gfx crucible tests
  • Add optee support tag 406
  • Add reboot testing for dss
  • Add gps module detection test
  • Add gpio loopback test
  • Add nvidia cuda test
  • Refactor dss tests using siblings
  • Automate wake-on-lan tests
  • Add checkbox-gfx opengl tests
  • Add groups to jobs
  • Add nvidia-drm test plan
  • Add new randr cycle script for wayland
  • Add a few vulkan smoke tests

Notable Bugfixes

  • This fix typeerror: fakejobrunner.run_job() got an unexpected keyword
  • Different names for different testplans
  • No more requests unixsocket
  • Add manifests for suspend_advanced_auto and multi-touch
  • Exporters:html: fix i/o log links when not-supported jobs exist
  • Force using fork multiprocessing
  • Completely deprecate the usage of gnome-randr package
  • Move site-packages to higher priority
  • Refactor light sensor test
  • Maken sleep key and suspend led jobs depend on suspend_advanced_auto
  • Update submission json schema to include system_information
  • Remove all jinja conditionals from requires
  • Only deactivate activated connections in wifi_nmcli_test.py
  • Remove all jinja conditionals from commands
  • Avoid collecting 80k lines of logs in checkbox collector
  • Add support for xhci-pci-renesas in the usb3 test.
  • Only run suspend-on-ac-power on devices with batteries
  • Fix: usb sd card reader support
  • Revise eddystone scanner
  • Update ov5645 test scenario
  • Added test_data folder to manifest
  • Metabox qol
  • Fix rvs confinement issues
  • Restores missing power management test plan in desktop 24-04
  • Ask the user to use headphones for both input/output in external headphone tests
  • Remove all (un)translatable warnings
  • Fix utitiles readiness for ce-oem-mipi-camera tests
  • Add manifest for ce-oem-gpio/loopback test
  • Provide checkbox_runtime environment variable during job execution only
  • Removed camera tests form core16 on rpi
  • [bugfix] remove uneeded nvdimm tests
  • Move some modules from checkbox_support to checkbox_ng
  • Remove screen-private-ppa test definition from contrib because it was added in base provider
  • Bump psf/black from 25.12.0 to 26.1.0
  • Try to give a more informative error as this is user facing
  • Fix wrong jobs that included in iot classic plan
  • Preserve-cwd only when preserve-cwd is provided
  • Specify fwts data directory for snap checkbox
  • Pulseaudio server utils not listing unused profiles
  • Drop only caps that were given to the process
  • Preserve nsenter cwd
  • Hardcode xdg_runtime_dir and dbus path
  • Lock focal version to 5.12 and transition all others to latest
  • Exclude sru disk stress tests from server-disk test plan
  • Fix fstrim test to not run when fstrim is not supported
  • Build static nsenter for core16
  • Fix documentation check issue with sitemap
  • Gnome monitor doesn’t return the active resolution
  • Move boot tests to the end of the iot automated test plan
  • Statically link nsenter and use nsenter.static
  • Fix bad path object in gl_support_test
  • Fix: the unavailable test not print the log
  • Bump actions/setup-python from 6.1.0 to 6.2.0
  • Purge fscrypt on <=18.04 as it doesn’t exist on ppc64el on 18.04
  • Add bluetooth/detect + after-suspend bluetooth/detect to bluetooth-cert-automated
  • Remove setpriv from dependencies
  • Use ambientcapabilities on non-core16 systems
  • Change location of stress-ng disk tests to avoid running on a tempfs
  • Enable broadcom sriov testing
  • Updated udevadm filter for capture devices
  • Dont use snap_user_common as it is not always normal-user readable
  • Allow systemd user unit to nsenter without sudo
  • Add intel granite rapids-d cpuid
  • Fix plenty typos in unit fields
  • Move “not supported tests” in their own section in the html report
  • Series16/snapcraft.yaml: remove fscrypt dependency
  • Remove legacy autorestart flag handling
  • Ci fix
  • Fix vulkan launchers
  • Add timeout to prevent udevadm stuck with faulty device
  • Fixed boolean in wwan tests
  • Append ld_library_path for lapack and blas
  • Use awk instead of cut for session filtering
  • Remove encoder_psnr test for qualcomm soc
  • Fixed detecting gps module by gpsctl
  • Fix gst discoverer variable
  • Use custom gst-discoverer binary
  • Include rpmsg tests into ce-oem test plan
  • Dmi sane_product missing mini pc and fix aio for related jobs
  • Remove bodge to disconnect double controller
  • Move boot status check to the beginning of reboot checks
  • Minimum resolution depends on a desktop session being available
  • Systemd based runner default on core
  • Add lan.* as a valid wired interface
  • Correct typo of serial echo server manifest
  • Hardcode systemd_ignore_chroot forced to 1
  • Urwid raw_display moved to display.raw
  • Service must be disabled by default
  • Fix minimium_resolution tests for ubuntu 25+
  • Add env var to ignore i²c buses in i²c test
  • Change the wifi 7 scripts to executable
  • Allow rocm-validation-suite deb in amd gpgpu test requires
  • Update instructions for rs232-console
  • Add while loop to check if restartcount is expected in 30 sec
  • Log run command
  • Modified cpufreq_governors.py
  • Restart the session instead of setupping
  • Fix: add more mock time values in test_prime_offload_tester to handle
  • Fix scripts relying on find_pkname_is_root_mountpoint
  • Make gst_utils.py be execuatble
  • Fix checkbox-support manifest
  • Fix multiple thermal devices path exist
  • Add manifest for disk/apste job
  • Fix net_driver_info crash when dumping parameters
  • Fix missing name field for nvme devices without devname
  • Use --non-interactive instead of -a
  • Ignore some partitions in udevadm parser
  • Fix: update odm and pc test plans to remove misc full plans and align automated/manual
  • Changed the image used for the attachment
  • Add “main server chassis” to not-portable sane_product for nvidia dgx spark
  • Use template-id to include disk_stress_ng in stress-ng-automated test plan
  • Fixed snap confinement issue with rocm-validation-suite
  • Add jobs dependency with suspend_advanced_auto
  • Revert “fixed eddystone_scanner fallback issue (new)”
  • Fix not-portable → non-portable typo
  • Remove large image capture from the rpi camera tests
1 Like