Checkbox 7.0.0 stable release

Project Discussion Checkbox
1

A new version of Checkbox is now available as a Debian package (in the Stable PPA ) and as Snaps (in the stable channel)!

Breaking changes

This release brings the new systemd-based test runner to the stable channel, on-by-default when using strict snaps. This is the culmination of our collaboration with the Snapd team to bring you better testing capabilities with strict snaps.

What Changes

The new runner spawns all of your tests outside of the AppArmor sandbox. This means not only that they are now exempt from it, but you can also now start to test actually test it. Before you were limited by the fact that being devmode, most infractions were allowed, while now, you can actually enter a snap context properly and see if it complies.

Other changes

This release brings substantial GPU and graphics testing improvements, including Vulkan smoke tests, Level Zero raytracing, NVIDIA CUDA and DRM test plans, and a migration to glmark2 for GL support verification. Hardware coverage expands with new tests for smartcards, barcode scanners, cash drawers, printers, PWM fans, and GPIO loopback, alongside WiFi MLO and automated Wake-on-LAN testing. On the infrastructure side, a new systemd-unit based runner and custom frontend interface offer more flexibility in how tests are executed and presented.

Reminder: if you update Checkbox you will not be able to resume testing session that you begun before updating. If this is undesirable to you, consider pinning Checkbox and finishing all sessions that are ongoing with the following command:

# for snaps:
$ sudo snap refresh --hold=forever checkbox checkbox24
# for debs:
$ sudo apt-mark hold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

Remember to eventually run the following to re-enable automatic updates:

# for snaps
$ sudo snap refresh --unhold checkbox checkbox24
# for debs
$ sudo apt-mark unhold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

New in this release

  • Support extra envvars per snap
  • Add smartcard test
  • Add cash drawer test
  • Add printer test
  • Add webgl conformance test
  • Test rpmsg function by checking each subsystem and device tree
  • Add barcode scanner test
  • Providers/base: add fscrypt test
  • Migrate to new randr cycle script
  • Audio server utils module
  • Update gl support test to use glmark2
  • Add codec test for koto
  • Migrate pc-sanity to new script for gpu drivers
  • Add imx8m video codec test
  • Add level zero raytracing testing
  • Extend camera tests for nxp imx8mp evk
  • Add pwm fan tests
  • Add the nvidia-requested vk cts subset
  • Add screen-private-ppa test
  • Systemd-unit based runner
  • Add serial stress test
  • Add a test case to verify the test audio i/o after suspend with previously occupied devices
  • Add bluetooth 6.0’s hci version in beacontools
  • Refactor bios error test script with python and add unit tests
  • Add a simple wifi mlo test
  • Custom frontend interface
  • Add level zero testing
  • Add running dss integration tests
  • Add checkbox-gfx crucible tests
  • Add optee support tag 406
  • Add reboot testing for dss
  • Add gps module detection test
  • Add gpio loopback test
  • Add nvidia cuda test
  • Refactor dss tests using siblings
  • Automate wake-on-lan tests
  • Add checkbox-gfx opengl tests
  • Add groups to jobs
  • Add nvidia-drm test plan
  • Add new randr cycle script for wayland
  • Add a few vulkan smoke tests

Notable Bugfixes

  • Exclude sru disk stress tests from server-disk test plan
  • Add test-snapd-tools-core24 support
  • Fix fstrim test to not run when fstrim is not supported
  • Build static nsenter for core16
  • Fix documentation check issue with sitemap
  • Gnome monitor doesn’t return the active resolution
  • Move boot tests to the end of the iot automated test plan
  • Statically link nsenter and use nsenter.static
  • Fix bad path object in gl_support_test
  • Fix: the unavailable test not print the log
  • Purge fscrypt on <=18.04 as it doesn’t exist on ppc64el on 18.04
  • Add bluetooth/detect + after-suspend bluetooth/detect to bluetooth-cert-automated
  • Remove setpriv from dependencies
  • Use ambientcapabilities on non-core16 systems
  • Change location of stress-ng disk tests to avoid running on a tempfs
  • Enable broadcom sriov testing
  • Updated udevadm filter for capture devices
  • Dont use snap_user_common as it is not always normal-user readable
  • Allow systemd user unit to nsenter without sudo
  • Add intel granite rapids-d cpuid
  • Fix plenty typos in unit fields
  • Move “not supported tests” in their own section in the html report
  • Series16/snapcraft.yaml: remove fscrypt dependency
  • Remove legacy autorestart flag handling
  • Fix vulkan launchers
  • Add timeout to prevent udevadm stuck with faulty device
  • Fixed boolean in wwan tests
  • Append ld_library_path for lapack and blas
  • Use awk instead of cut for session filtering
  • Remove encoder_psnr test for qualcomm soc
  • Fixed detecting gps module by gpsctl
  • Fix gst discoverer variable
  • Use custom gst-discoverer binary
  • Include rpmsg tests into ce-oem test plan
  • Dmi sane_product missing mini pc and fix aio for related jobs
  • Remove bodge to disconnect double controller
  • Move boot status check to the beginning of reboot checks
  • Minimum resolution depends on a desktop session being available
  • Systemd based runner default on core
  • Add lan.* as a valid wired interface
  • Correct typo of serial echo server manifest
  • Hardcode systemd_ignore_chroot forced to 1
  • Urwid raw_display moved to display.raw
  • Service must be disabled by default
  • Fix minimium_resolution tests for ubuntu 25+
  • Add env var to ignore i²c buses in i²c test
  • Change the wifi 7 scripts to executable
  • Allow rocm-validation-suite deb in amd gpgpu test requires
  • Update instructions for rs232-console
  • Add while loop to check if restartcount is expected in 30 sec
  • Log run command
  • Modified cpufreq_governors.py
  • Restart the session instead of setupping
  • Fix: add more mock time values in test_prime_offload_tester to handle
  • Fix scripts relying on find_pkname_is_root_mountpoint
  • Make gst_utils.py be execuatble
  • Fix checkbox-support manifest
  • Fix multiple thermal devices path exist
  • Add manifest for disk/apste job
  • Fix net_driver_info crash when dumping parameters
  • Fix missing name field for nvme devices without devname
  • Use --non-interactive instead of -a
  • Ignore some partitions in udevadm parser
  • Fix: update odm and pc test plans to remove misc full plans and align automated/manual
  • Changed the image used for the attachment
  • Add “main server chassis” to not-portable sane_product for nvidia dgx spark
  • Use template-id to include disk_stress_ng in stress-ng-automated test plan
  • Fixed snap confinement issue with rocm-validation-suite
  • Add jobs dependency with suspend_advanced_auto
  • Revert “fixed eddystone_scanner fallback issue (new)”
  • Fix not-portable → non-portable typo
  • Remove large image capture from the rpi camera tests