Upstream bug LP:#2004555 (CVE-2023-2088) identified that “an unauthorised access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host”. This is tracked in Ubuntu packaging via bug LP:#2021980.
While the bug affects iSCSI and FC, the fix introduces a breaking change for all configurations upgrading to the fixed packages if they do not have service token support enabled. The new code introduces mandatory use of service tokens to identify requests coming from a trusted control plane service. Therefore, regardless of the cinder backend in use, service tokens must be enabled prior to upgrading to the fixed packages.
Service tokens are issued by Keystone for service users. For example, a token can be issued to the Nova service user. They have a particular trust level, defined by the service, that allows them to perform trusted actions. In the case of Nova, the action is detaching volumes from instances.
Versions of Ubuntu OpenStack for Yoga and later will have new packages versions released with CVE patches applied that require the use of service tokens to enable the detaching and deletion of volumes on instances. These packages will be released on July 24, 2023.
Versions of Ubuntu OpenStack for Xena and earlier will not have package updates released and will still be vulnerable to the “unauthorised detach of volumes” issue until a mitigation policy is applied. The mitigation policy provided requires the use of service tokens.
For Charmed OpenStack releases, this means that some charms (identified below) must be upgraded to enable service tokens prior to the packages being upgraded and before the mitigation policy is applied. If not, users will lose the ability to detach volumes from their instances. It is important to note that this affects all cinder backends.
Patched packages
The following package versions include patches for CVE-2023-2088. These package versions deliver services that require the use of service tokens. Package versions prior to those listed will not be patched for CVE-2023-2088 and will therefore require policy updates for mitigation.
Ubuntu 23.04 (Lunar)
- nova >= 3:27.0.0-0ubuntu1.3
- cinder >= 2:22.0.0-0ubuntu1.3
- python-os-brick >= 6.2.0-0ubuntu2.3
- ironic >= 1:21.4.0-0ubuntu1.1
- python-glance-store >= 4.3.0-0ubuntu1.3
Ubuntu Cloud Archive Antelope
- nova >= 3:27.0.0-0ubuntu1.3~cloud0
- cinder >= 2:22.0.0-0ubuntu1.3~cloud0
- python-os-brick >= 6.2.0-0ubuntu2.3~cloud0
- ironic >= 1:21.4.0-0ubuntu1.1~cloud0
- python-glance-store >= 4.3.0-0ubuntu1.3~cloud0
Ubuntu 22.10 (Kinetic)
- nova >= 3:26.1.1-0ubuntu1.1
- cinder >= 2:21.2.0-0ubuntu1.1
- python-os-brick >= os-brick - 6.1.0-0ubuntu1.3
- ironic >= 1:21.1.0-0ubuntu1.1
- python-glance-store >= 4.1.0-0ubuntu1.3
Ubuntu Cloud Archive Zed
- nova >= 3:26.1.1-0ubuntu1.1~cloud0
- cinder >= 2:21.2.0-0ubuntu1.1~cloud0
- python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0
- ironic >= 1:21.1.0-0ubuntu1.1~cloud0
- python-glance-store >= 4.1.0-0ubuntu1.3~cloud0
Ubuntu 22.04 (Jammy)
- nova >= 3:25.1.1-0ubuntu1.1
- cinder >= 2:20.2.0-0ubuntu1.1
- python-os-brick >= 5.2.2-0ubuntu1.2
- ironic >= 1:20.1.0-0ubuntu1.1
- python-glance-store >= 3.0.0-0ubuntu1.3
Ubuntu Cloud Archive Yoga
- nova >= 3:25.1.1-0ubuntu1.1~cloud0
- cinder >= 2:20.2.0-0ubuntu1.1~cloud0
- python-os-brick >= 5.2.2-0ubuntu1.2~cloud0
- ironic >= 1:20.1.0-0ubuntu1.1~cloud0
- python-glance-store >= 3.0.0-0ubuntu1.3~cloud0
Charms with Service Tokens Enabled by Default
Following is the list of charms and earliest revision that supports service tokens:
Channel: 2023.1/stable
- Keystone revision >= 645
- Cinder revision >= 639
- Nova-compute revision >= 678
- Nova-cloud-controller revision >= 674
- Ironic-api revision >= 54
- Ironic-conductor
- s390x revision >= 151
- arm64 revision >= 152
- ppc64el revision >= 153
- amd64 revision >= 149
Channel: zed/stable
- Keystone revision >= 646
- Cinder revision >= 647
- Nova-compute revision >= 683
- Nova-cloud-controller revision >= 676
- Ironic-api revision >= 53
- Ironic-conductor
- s390x revision >= 148
- arm64 revision >= 154
- ppc64el revision >= 150
- amd64 revision >= 147
Channel: yoga/stable
- Keystone revision >= 647
- Cinder revision >= 638
- Nova-compute revision >= 681
- Nova-cloud-controller revision >= 679
- Ironic-api revision >= 48
- Ironic-conductor revision >= 145
Channel: xena/stable
- Keystone revision >= 651
- Cinder revision >= 640
- Nova-compute revision >= 680
- Nova-cloud-controller revision >= 677
- Ironic-api revision >= 50
- Ironic-conductor revision >= 143
Channel: wallaby/stable
- Keystone revision >= 653
- Cinder revision >= 648
- Nova-compute revision >= 684
- Nova-cloud-controller revision >= 675
- Ironic-api revision >= 52
- Ironic-conductor revision >= 144
Channel: victoria/stable
- Keystone revision >= 650
- Cinder revision >= 646
- Nova-compute revision >= 679
- Nova-cloud-controller revision >= 678
- Ironic-api revision >= 49
- Ironic-conductor revision >= 142
Channel: ussuri/stable
- Keystone revision >= 652
- Cinder revision >= 645
- Nova-compute revision >= 682
- Nova-cloud-controller revision >= 680
- Ironic-api revision >= 51
- Ironic-conductor revision >= 155
Using a custom Cinder policy as a mitigation
It’s possible to restrict the access to the operations affected by this CVE by providing a custom Cinder policy file that will prevent users from deleting attachments, detach and force detach. In order to use this mitigation, service token support must be enabled. The exact method for customising this depends on the deployment tooling being used. For Charmed OpenStack users it is the following:
- Identify the nova service user id running the following command
openstack user show --domain service_domain -f value -c id nova
The command output looks like this
ae034831d6bf41099327bf9d34988fbc
- Create a policy file named cve-2023-2088-cinder.yaml with the following content:
“is_service”: “role:service or service_user_id:<nova_service_uuid>”
“legacy_system_admin_or_project_member”: “(role:admin) or (role:member and project_id:%(project_id)s)”
“volume:attachment_delete”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_actions:terminate_connection”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_actions:detach”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_admin_actions:force_detach”: “!”
Replacing “<nova_service_uuid>” would result in a snippet that looks like this:
“is_service”: “role:service or service_user_id:ae034831d6bf41099327bf9d34988fbc”
“legacy_system_admin_or_project_member”: “(role:admin) or (role:member and project_id:%(project_id)s)”
“volume:attachment_delete”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_actions:terminate_connection”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_actions:detach”: “rule:legacy_system_admin_or_project_member and rule:is_service”
“volume_extension:volume_admin_actions:force_detach”: “!”
- Create zip files from the yaml file:
zip cve-2023-2088-cinder.zip cve-2023-2088-cinder.yaml
- Attach the zip file as a resource to the cinder applications:
juju attach-resource cinder policyd-override=cve-2023-2088-cinder.zip
- Enable the overrides via the use-policyd-override charm option:
juju config cinder use-policyd-override=true
For more details on using policy overrides see the OpenStack Charm Guide.
Frequently Asked Questions
Q: What can I do if my deployed cloud doesn’t have service tokens enabled and my deployment tooling doesn’t have the option to enable them?
A: Pin the version of the affected packages to block the upgrades to the versions listed in “Affected packages” section or newer, until service tokens are enabled.
Q: For Charmed OpenStack, what’s the upgrade order for charms and deb packages?
A: Upgrade the charms to the latest versions available in the charm channel tracked by your environment or at the least to the revisions listed in section “Charms with Service Token Enabled By Default”. After that, upgrade the deb packages with your preferred method (e.g. Landscape, apt-get, etc).
References
- CVE-2023-2088: CVE-2023-2088 | Ubuntu
- OSSA-2023-003 Unauthorized volume access through deleted volume attachments: https://security.openstack.org/ossa/OSSA-2023-003.html
- OSSN-0092 Using Configuration as a Short-Term Mitigation for OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092
- Upstream Bug 2004555: Bug #2004555 “[OSSA-2023-003] Unauthorized volume access through...” : Bugs : OpenStack Compute (nova)
- Ubuntu Packaging Bug 2021980: Bug #2021980 “Unauthorized volume access through deleted volume ...” : Bugs : Ubuntu Cloud Archive
- OpenStack Charm Guide Policy overrides: https://docs.openstack.org/charm-guide/latest/admin/policy-overrides.html