Checkbox 4.0.0 beta release

Certification Checkbox
1

Hello all!

Checkbox v4.0.0 is now available as a Debian package (in the Beta PPA) and as Snaps (in the beta channel)! We have made several modifications to Checkbox in this last period, most notably we have focused on many aspects of resuming a testing session via remote. Checkbox is now able to resume remote sessions both manually, via a new URWID menu, and automatically. This was needed because a Checkbox session can be interrupted by a user (or a reboot/crash of the device under test). Until today, it was complicated to resume a session that was manually interrupted and the automatic resume did not cover all possible situations where it was needed. This required the introduction of a new remote API so it is a breaking change: do upgrade both the Agent and the Controller or neither!.

Reminder: if you update Checkbox you will not be able to resume testing session that you begun before updating. If this is undesirable to you, consider pinning Checkbox and finishing all sessions that are ongoing with the following command:

# for snaps:
$ sudo snap refresh --hold=forever checkbox checkbox22
# for debs:
$ sudo apt-mark hold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

Remember to eventually run the following to re-enable automatic updates:

# for snaps
$ sudo snap refresh --unhold checkbox checkbox22
# for debs 
$ sudo apt-mark unhold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

Breaking Change:

With this version, we introduce one breaking change. We upgraded the remote API. What this means is that any remote couple (Agent, Controller) must be at the same >=4.0.0 version to function together. If you are unsure about the implications of this change, please refer to this documentation page . If you have any more questions or you need support you can always find us on Mattermost or on Discourse.

Preview of the resume menu

If you have any resumable session, Checkbox will present you the option of resuming via the new menu in the test plan selection screen (R):

image
image1064×262 35.5 KB

Pressing it will present you the following menu with a list of all resumable sessions and a preview of the selected session information:

image
image1284×409 45.4 KB

Selecting a session via (Enter) will present you a sub-menu that allows you to comment on the last running job, set its result or simply re-run it:

image
image1284×409 54.2 KB

New in this release:

  • Ensure that jobs that have an “after suspend” counterpart are run before suspend
  • Autoresume checkbox local
  • Refactor cpufreq governors jobs
  • Allow template ids to be used in Test Plans
  • Autoresume crashed/norerun sessions on interactive runs for remote
  • Opencv build from source on snaps
  • Resume screen for checkbox remote
  • Add sd card test in strict confinement mode
  • Add keys to fast interact with resume menu action
  • Enable multi interface testing in gateway_ping_test.py
  • Add template-id field to instantiated jobs
  • Add template-summary and template-description fields to the Template Unit
  • Add new cpuid identification (Siena, Emerald Lake)
  • Migrate pulseaudio to pipewire
  • Create Contrib area and add the ce-oem provider in it
  • New way to resume a Checkbox session via Urwid UI
  • Add “Hygon Dhyana” to CPUs recognized by cpuid.py + small refactor
  • Enhance version handling in packaging
  • Add certification-status field in list-bootstrapped command
  • Retire pm_test.py from warm/cold boot nested parts in desktop auto test plan
  • Add QEP test cases and plans from iiotg provider
  • Import ISHTP and ECLITE test cases and plans from iiotg provider
  • Add VA API test case
  • Collect system information
  • Add script and resource job to return information about classic and core images
  • New test case for core.efi file check needed by grub and shim upgrade
  • Retire pm_test from PC suspend 30 cycles and separate IoT suspend cycles
  • Add snap refresh/revert jobs to SRU test plans and add manifest entries to control when to execute them
  • Ubuntu 24.04 & Core 24 test plans
  • Add miscellanea/ubuntu-desktop-minimal-recommends job

Notable Bugfixes:

  • Remote resistant no launcher
  • Support job flags being None
  • Allow listing instances from resume state
  • Removed unused screenshot validation test and related file
  • Cast back exception to local type
  • Strip color code in unity_support_test results
  • Fix resuming local manual jobs
  • Revert libde265 change on armhf
  • [checkbox-ce-oem] Fix wrong arg name
  • Support different kernel repo
  • Fixed failed ping test
  • Be verbose when breaking pipes when interacting with bluetoothctl
  • Support case insensitive searching in names or ids
  • Refactor LED tests
  • ignore dm devices that serve only ubuntu-* partitions
  • Remove $SNAP_DATA refs from checkbox-support UT
  • Removed unused screenshot tests
  • Fix gateway ping
  • Fix flag name for cpu/topology
  • Fix quit option for manual jobs
  • Add uniqueness for Template Units id
  • Update the Virtualization Script for 24.04
  • Explicitly use verbose kwarg
  • Fix and refactor gateway_ping_test
  • Improve suspend/suspend_advanced* tests #857
  • Update many missing categories
  • Support user defined snaps from config_var to fix issue 923
  • Fix crash on missing plainbox-provider-develop
  • Update ping test to not crash on failing pings
  • Add missing category to resource jobs
  • Mock module and explain why it was done
  • Changed variable name to environment from environ
  • Update contruct to v2.10.57
  • Renamed packaging.py to packaging_metadata.py
  • Rename ARM Vector Floating Point templates to avoid clash
  • Fixed SyntaxWarning correctly marking regex strings
  • Fix unknown function in tests for checkbox-support
  • Set inxi permissions (+x) in checkbox-ng debian packaging
  • Fix the wrong file name from pactl.sh to pactl_list.sh
  • Update the rotation test cases to against the bug
  • Fix disk resource job to fetch SMART status
  • Unique validator for packaging metadata
  • fix detecting insertion/removal of tb3 drive
  • Adjust some test sequence to make the manual tests a little easy
  • Added a checkbox-shell flag
  • Update the Virtualization Test to Clean up Logs
  • Fix snap boot kernel path
  • Add executable flag to grub_file_check.sh
  • fprintd should enroll a user, not a uid
  • Modify detection method for usb-dwc3 and provide more info when it fails
  • Make network-info jobs require ethtool to be available
  • Don’t use ‘-A’ in network.py on non-numa systems
  • Check the installed package instead of from ubuntu-archive
  • virtualization/verify_lxd_vm requires lxd versions only available as snaps
  • update the search pattern for CAN device
  • Adjust the wwan, watchdog and alsa-loopback test cases.
  • Move memory_stress_ng from “memory” category to “stress” category
  • fix mocks in tests for remote_session_assistant
  • Add/fix resource debugging and a tests for #665
  • Refactor bt_list_adapters.py and add unit tests to it
  • Always resume automatic sessions on remote
  • Remove reference to bin when importing modules in some unit tests
  • Fix: wireless_scanning needs to be run as root while run it remotely.
  • [checkbox-ce-oem] fixed CAN bus and serial RS485 bug
3 Likes