Checkbox 4.0.0 stable release

Project Discussion Certification
1

Hello all!

Checkbox v4.0.0 is now (finally!) available as a Debian package (in the Stable PPA ) and as Snaps (in the stable channel)!

We have made several modifications to Checkbox in this last period, most notably we have focused on many aspects of resuming a testing session via remote. Checkbox is now able to resume remote sessions both manually, via a new Urwid menu, and automatically. This was needed because a Checkbox session can be interrupted by a user (or a reboot/crash of the device under test). Until today, it was complicated to resume a session that was manually interrupted and the automatic resume did not cover all possible situations where it was needed. This required the introduction of a new remote API so it is a breaking change: do upgrade both the Agent and the Controller or neither!.

Compared to 4.0.0 beta release, a lot of things were updated to make sure this new feature worked in all the configurations our users may encounter.

Reminder: if you update Checkbox you will not be able to resume testing session that you begun before updating. If this is undesirable to you, consider pinning Checkbox and finishing all sessions that are ongoing with the following command:

# for snaps:
$ sudo snap refresh --hold=forever checkbox checkbox22
# for debs:
$ sudo apt-mark hold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

Remember to eventually run the following to re-enable automatic updates:

# for snaps
$ sudo snap refresh --unhold checkbox checkbox22
# for debs 
$ sudo apt-mark unhold checkbox-ng checkbox-provider-resource checkbox-provider-certification-client checkbox-provider-base

A big thanks to people who made their first contributions for this release:

New in this release

  • Resume Screen for Checkbox Remote expose API (as explained in the post about the 4.0.0 beta release)
  • Add miscellanea/ubuntu-desktop-minimal-recommends job
  • Ubuntu 24.04 & Core 24 test plans
  • Manifest snap update jobs
  • Retire pm_test from PC suspend 30 cycles and separate IoT suspend cycles
  • Add test_image_checker in to checkbox_support to check the image
  • Collect system information
  • Add VA API test cases from checkbox-providers
  • Move ISHTP and ECLITE test cases and plans from iiotg provider and checkbox-iiotg-provder
  • Add QEP test cases and plans from checkbox-iiotg-providers and iiotg provider
  • Retire pm_test.py from warm/cold boot nested parts in desktop auto test plan
  • Add: include certification-status fields
  • Enhance version handling in packaging
  • New way to resume a Checkbox session via Urwid UI
  • Create Contrib area and add the ce-oem provider in it
  • Migrate pulseaudio to pipewire
  • Add new cpuid identification (Siena, Emerald Lake)
  • Add template-summary and template-description fields to the Template Unit
  • Add template-id field to instantiated jobs
  • Enable multi interface testing in gateway_ping_test.py
  • Add keys to fast interact with resume menu action
  • Add sd card test in strict confinement mode
  • Resume screen for checkbox remote
  • Opencv build from source on snaps
  • Autoresume crashed/norerun sessions on interactive runs for remote
  • Allow template ids to be used in Test Plans
  • Refactor cpufreq governors jobs
  • Autoresume checkbox local
  • Ensure that jobs that have an after suspend counterpart are run before suspend
  • Cover smaller genio tests
  • Cover cpu idle genio
  • Cover boot partition genio
  • [checkbox-ce-oem] Make serial test (console and transmit data) automated
  • Cover brightness genio test
  • Cover loopback genio test
  • New timeout helper module
  • Add tiny bit of tests to the storage tests
  • [checkbox-ce-oem] add iio sensors test
  • Add MEI test cases and plans from checkbox-iiotg-provider and iiotg provider
  • [checkbox-ce-oem] Add thermal test in strict confinement mode(New)
  • Add ‘expand’ subcommand to list jobs and templates in a test plan
  • Add realtime performance test
  • Update checkbox.json to include template_id
  • Moved genio from contrib to providers
  • Move checkbox-ce-oem snapcraft.yaml into contrib area
  • Add checking RAID mode test
  • Add submision for camera quality test
  • [checkbox-ce-oem] Restructure checkbox ce oem
  • Switching power mode for Lenovo’s platforms
  • Add systemd-analyze critical-chain job
  • Adding rt test plan
  • Add fwupdmgr attachment job
  • Created launchers for 24.04 Release
  • [checkbox-ce-oem] Add GPIO slot test for gadget SNAP
  • Include obex test into bluez-automated test plan
  • Improve the 24.04 IoT Cert Test Plans
  • Add Gstreamer video decoder md5 checksum scenario
  • Add remote processor firmware loading test
  • [checkbox-ce-oem] Add timedatectl test in strict-confinement mode
  • Fix id: graphics/{index}gl_support{product_slug} alway return ture
  • Add camera submission for multiple resolution test
  • Add sysfs led indicator to auto coverage
  • Migrate pc-santiy to contrib
  • Add disconnect function to snapd class
  • Add platform-userguide-test
  • add qep, edac, mei, ishtp and va-api into IoT test plans and ODM test plans
  • Refactor WIFI STAs resource job and Support Wifi 7
  • [checkbox-ce-oem] Add SPI Test
  • Add WIFI 7 (BE) Test Case and Plan
  • Automate DVD test

Notable Bugfixes

  • Fix metabox (apt profile + daemon respawn)
  • Fix: wireless_scanning needs to be run as root while run it remotely.
  • Remove reference to bin when importing modules in some unit tests
  • Always resume automatic sessions on remote
  • Refactor bt_list_adapters.py and add unit tests to it
  • Pin urllib to <2.0 for Metabox
  • Add/fix resource debugging and a tests for #665
  • fix mocks in tests for remote_session_assistant
  • Move memory_stress_ng from the category memory to stress
  • Adjust the wwan, watchdog and alsa-loopback test cases.
  • update the search pattern for CAN device
  • virtualization/verify_lxd_vm requires lxd versions only available as snaps
  • Check the installed package instead of from ubuntu-archive
  • Don’t use ‘-A’ in network.py on non-numa systems
  • fix: make network-info jobs require ethtool to be available
  • Modify detection for usb-dwc3 to make it really detecting
  • fprintd should enroll a user, not a uid
  • Add executable flag to grub_file_check.sh
  • Fix snap boot kernel path
  • Update the Virtualization Test to Clean up Logs
  • Added a checkbox-shell flag
  • Adjust some test sequence to make the manual tests a little easy
  • fix detecting insertion/removal of tb3 drive
  • Unique validator for packaging metadata
  • Fix disk resource job to fetch SMART status
  • Update the rotation test cases to against the bug
  • fix the wrong file name pactl.sh to pactl_list.sh
  • Set inxi permissions (+x) in checkbox-ng debian packaging
  • Fix unknown function in tests for checkbox-support
  • Fixed SyntaxWarning correctly marking regex strings
  • Rename ARM Vector Floating Point templates to avoid clash
  • Renamed packaging.py to packaging_metadata.py
  • Update contruct to v2.10.57
  • Changed variable name to environment from environ
  • Mock module and explain why it was done
  • Fix the wrong category issue for the camera 5986:118c
  • Add missing category to resource jobs
  • Update ping test to not crash on failing pings
  • Fix crash on missing plainbox-provider-develop
  • Support user defined snaps from config_var to fix issue 923
  • Update many missing categories
  • Improve suspend/suspend_advanced* tests #857
  • Fix and refactor gateway_ping_test
  • Explicitly use verbose kwarg
  • Update the Virtualization Script for 24.04
  • Add uniqueness for Template Units id
  • Fix quit option for manual jobs
  • Fix flag name for cpu/topology
  • Fix gateway ping
  • Removed unused screenshot tests
  • Remove $SNAP_DATA refs from checkbox-support UT
  • ignore dm devices that serve only ubuntu-* partitions
  • Refactor LED tests
  • Support case insensitive searching in names or ids
  • Be verbose when breaking pipes when interacting with bluetoothctl
  • Fixed failed ping test
  • Support different kernel repo
  • [checkbox-ce-oem] Fix wrong arg name
  • Revert libde265 change on armhf
  • Fix resuming local manual jobs
  • Strip color code in unity_support_test results
  • Cast back exception to local type
  • Removed unused screenshot validation test and related file
  • Allow listing instances from resume state
  • Support job flags being None
  • Remote resistant no launcher
  • [checkbox-ce-oem] fixed CAN bus and serial RS485 bug
  • Fix dkms build validation in mantic
  • Remove snappy-revert sub-tp from sru
  • Add TemplateUnit.repr and modify its str method
  • Apply overrides even when using template id in test plan definition
  • Don’t diff results of the network test
  • Replace create with up --no-start in docker tests
  • fix the name of the Alder Lake codename
  • Fix feild name for scaling_test and maxfreq_test log attach from after to depeneds
  • Re-introduce wireless/check_iwlwifi_microcode_crash_{interface} template
  • Clearer autoresume text
  • Correct the suspend/resume threshold variable
  • Remove the dependency on Sylvain’s ppa to get libraspberrypi0
  • Change to new graphic test strategy
  • Add requires for bluetooth/keyboard and bluetooth/keyboard-manual
  • update definitions in base/audio jobs using an AI tool
  • Added 10m timeout to eddystone bluetooth test
  • Take inline overrides into account in mandatory_include and bootstrap_include sections of a test plan
  • Fix erroneous additions to PXUs
  • Add ‘environ’ for the job ‘graphics/{index}glmark2-es2{product_slug}’ and change from automated to manual
  • [Checkbox-ce-oem] Modify job and script to support use x-test tool(Bugfix)
  • Fix resume forgetting previous job outcome
  • Updated the summary of DI/O tests and ce oem test plan
  • Replacing value of name to direction in logic to make this validation more robust
  • [checkbox-ce-oem] Fix serial tests
  • Refactor snapd_resource.Features methods
  • Test genio pipeline
  • Move lsb_release install up to stage and build of checkbox-support
  • checkbox-support: add missing pyyaml dependency
  • Fix config loading after resume
  • Set default argv
  • change the naming of check_audio_deamon.sh to check_audio_daemon.sh
  • migrate keys/microphone-mute from pulseaudio to pipewire
  • [checkbox-ce-oem] Fix jobs and testplans to pass validate
  • Add audio manifest entries for audio ports and internal speakers
  • Fix config loading local when resumable session available
  • [checkbox-ce-oem] Revise snapcraft.yaml to build successfully
  • Disk resource gated jobs
  • Added udisks2 requirement
  • Correct the alsa_info user
  • Correct desktop 2404 suspend cycles plan
  • Retire hwclock
  • blocker/non-blocker status added to a few tests
  • Fix the audio_after_suspend test failing on mantic
  • exclude debsum test from the SRU testplan.
  • Remove many warnings
  • [checkbox-ce-oem] Fix path for checkbox-ce-oem workflow
  • Fix timeout children
  • Change to new graphic test strategy - jobs and test plans
  • New mock_timeout
  • Make sure the device reboots after snap refresh/revert tests
  • Fix the audio_after_suspend test failing on mantic - test case
  • Bring back kernel/snapd/gadget snaps revert/refresh jobs to the SRU test plan
  • Fix udevadm.py to parser VRAID partition correctly
  • Modify bluetooth/bluetooth_obex_send job able to run on both classic and core
  • Fixed bug when missing one item in the status and added debug info
  • Require one of the lxd or lxd-installer debs or the lxd snap for the virtualization tests.
  • Limit the platform to perform the sleepgraph test
  • Fix rpmsg tests
  • Fix spelling mistake in tutorial
  • Fall back from distutils → packaging.version
  • Fix checkbox remote running memory stress test
  • remove the unused [comment] section from config file
  • Fix modprobe_attachment and sysctl_attachment failed in remote mode
  • removed requirement of udisks2 on core
  • Fix the wrong output from suspend cycles
  • Make networking/http test more resilient
  • Fix base provider’s unit tests
  • Remove iot-cert-image-automated from ODM Server Test Plan
  • [checkbox-ce-oem] support raritan PDU for cold-reboot-pdu test
  • Fix the rest of job defintions
  • Get the Zapper keyboard device path dinamically
  • Fix wpa3 netplan test
  • Added SSL check to checkbox support
  • Remove pm-graph from SRU test plans
  • Fix: Make checkbox-ce-oem uc20 as a runnable checkbox
  • [chechkbox-ce-oem] Fix checkbox validate
  • [checkbox-ce-oem] Modify tcpecho_stress script
  • Included firmware revision and hw revision for wwan resource job
  • [checkbox-ce-oem] Add more detail for ce-oem-mtd/chec-total-numbers
  • Fix id: graphics/{index}gl_support{product_slug} alway return ture
  • Fix the issue networking_info_.* always run after suspend
  • Add requires to ce-oem-gpio check-slots job
  • Unplug the Zapper monitor at the end of the EDID test
  • Fix and test unpickleable case
  • Pick patch from old pc-sanity repo
  • Fix the pc-sanity install path
  • Carry over all QT variables in child process
  • Seperate check crypto profile tests
  • Fix the c binary install path in pc-sanity
  • Add environ field
  • Replace category == ‘AUDIO’ with has_audio_playback
  • Replaced open(Path, …) with Path.open(…)
  • Remove double install of the wrong package
  • plainbox-provider-pc-sanity (noble): depends openscap-scanner as well
  • Return error on timeout
  • netplan bugfix declare auth for non wpa3
  • Add missing checkbox dependency to pc-sanity
  • Added core24 to the list of the snaps that should be installed
  • Remove poitless job from canary
  • Remove hdapsd packaging dependency from base provider
  • [checkbox-ce-oem] Remove manifest from resource job
1 Like