Thanks Varshi! @ilvipero I wonder if you have a Summit organising checklist that you could add an item to please, to request Matrix maintenance window avoidance from Varshi’s team during that time?
Hello,
Please be advised that our Matrix service will undergo a planned upgrade on May 5, 2025 (next Monday) at 9:00 PM UTC.
The upgrade involves moving Synapse from version 1.127.0 to 1.128.0, which includes database schema changes.
During the upgrade window (approximately one hour), the service may become temporarily unavailable.
Because this upgrade modifies the database, please be aware that a recovery procedure will be required in case of failure. We are taking all necessary precautions to ensure a smooth process.
We appreciate your understanding and will keep you informed throughout the process.
5 Likes
Hello,
Please be advised that our Matrix service will undergo a planned upgrade on June 16, 2025 (next Monday) at 9:00 PM UTC.
The upgrade involves moving Synapse from version 1.128.0 to 1.130.0.
During the upgrade window (approximately one hour), the service may become temporarily unavailable.
Since there are no database changes, we expect a smooth process. However, please be aware that a recovery procedure will be required in case of failure.
We appreciate your understanding and will keep you informed throughout the process.
3 Likes
Hello everyone,
In response to a security pre-disclosure (see official announcement), we will be applying a critical security patch to our Matrix service on Tuesday, July 22, 2025.
Please note: the security update will require online maintenance that may affect room ownership. Specifically, all public rooms in the Ubuntu Space will have their “creator” switched to the moderator account, as a result of the way the vulnerability has been addressed.
To prepare for this, a preparatory upgrade is required tomorrow, July 18, 2025 at 9:00 PM UTC.
This upgrade involves moving Synapse from version 1.131.0 to 1.134.0.
During the upgrade window (approximately one hour), the service may become temporarily unavailable.
As there are no database changes involved, we expect the process to go smoothly. However, please note that a recovery procedure will be required in case of any failure.
We appreciate your understanding and will keep you informed throughout the process.
2 Likes
Hello everyone,
Following up on our previous communication, the preparatory upgrade originally scheduled for today has been rescheduled to Tuesday, July 22, 2025, at 9:00 PM UTC, as the security patch will be released on Monday, August 11, 2025.
We appreciate your understanding and will keep you informed throughout the process.
3 Likes
Hello everyone,
I’ve been investigating with some users issues after every maintenance. The symptoms are that the encrypted rooms / messages can’t be decrypted anymore and requires recovery of the keys from other devices. Another symptom is that “Set up Secure Backup” fails.
This issue seems to come from a corrupted Key store on server side and requires to reset it.
PLEASE READ INSTRUCTIONS BELOW BEFORE DOING ANYTHING WITH YOUR KEY STORAGE
Before doing any action on your key storage, we need to ensure you’re safe and won’t lose them (which will result in unrecoverable room messages).
- Ensure the device on which you’re doing the manipulation is cross-verified
- Make sure to have another device connected and verified
- Make sure you have exported your recovery key
Once all this is done you can reset your key storage:
- Go to Settings > Encryption
- Disable “Allow key storage”
- Accept the warning by clicking on “Delete key storage”
- Re-enable “Allow key storage”
- Proceed with “Set up Secure Backup”
Now to check if everything is fine:
- Go to “Settings > Help & About > Submit debug logs”
- Click on “Download logs”
- Open the
rageshake.tarfile download, and checkissue.txt - Search for
secret_storage_ready,session_backup_key_in_secret_storage,cross_signing_privkey_in_secret_storageandcross_signing_ready
They should all be true
1 Like
Upcoming Public Room Upgrades — What to Expect
Update (11/08/2025): Today we’ll be only applying the patch, the rooms upgrade will be planning and announced it here in this post.
As part of the security patch announced by the Matrix.org Security Team, we will be upgrading all public rooms on our server to the newly released room version 12. This upgrade is necessary to address critical vulnerabilities in the federation protocol and ensure that our communications remain secure.
What does this mean for you?
Due to how Matrix handles room upgrades under the hood, this process is not a seamless change. it is effectively:
create a new room → copy history and members → mark the old room as inactive
As a result, when you visit an upgraded room, you will see a banner message saying:
“This room has been replaced and is no longer active. The conversation continues here.”
Clicking “here” will take you to the new upgraded room where all future discussions will happen.
Timeline
We plan to start applying the security patch and this process on Monday, August 11th, 2025at 9:00 PM UTC.
Why this is necessary
This is part of a coordinated response to security vulnerabilities that could potentially affect any server in the public federation.
The upgrade to room version 12 is the only way to ensure that our rooms are protected under the updated protocol.
You can read more details in the official Matrix announcement:
https://matrix.org/blog/2025/07/security-predisclosure/
We know this user experience isn’t ideal, but it is the only secure option available at this time.
Thanks for your patience and understanding while we keep our communications safe.
2 Likes
Hello,
The upgrade has been rescheduled due to external reasons.
Please be advised that our Matrix service will undergo a planned upgrade on September 22 24, 2025 (next Monday Wednesday) at 9:00 PM UTC.
The upgrade involves moving Synapse from version 1.135.2 to 1.138.0.
During the upgrade window (approximately one hour), the service may become temporarily unavailable.
Since there are no database changes, we expect a smooth process. However, please be aware that a recovery procedure will be required in case of failure.
We appreciate your understanding and will keep you informed throughout the process.
3 Likes
Hello,
Our Matrix service will undergo a planned cluster migration on Monday, October 15, 2025, at 4:00 PM UTC.
This migration involves moving Synapse from one Kubernetes cluster to another, on a different network. It will continue pointing to the same database, but many aspects need to be tested before the service can be considered fully available again.
Unlike previous upgrades, this process is expected to take approximately 5 hours. We appreciate your understanding and will provide updates throughout the migration. In case of any issues, we will share details of the recovery process in this post.
If this schedule conflicts with any important events, please let us know.
2 Likes
Upcoming Public Room Upgrades — What to Expect
As part of the security patch announced by the Matrix.org Security Team, we will be upgrading all public rooms on our server to the newly released room version 12. This upgrade is necessary to address critical vulnerabilities in the federation protocol and ensure that our communications remain secure.
What does this mean for you?
Due to how Matrix handles room upgrades under the hood, this process is not a seamless change. it is effectively:
create a new room → copy history and members → mark the old room as inactive
As a result, when you visit an upgraded room, you will see a banner message saying:
“This room has been replaced and is no longer active. The conversation continues here.”
Clicking “here” will take you to the new upgraded room where all future discussions will happen.
Timeline
We plan to start applying the security patch and this process on Wednesday, November 26th, 2025 at 9:00 PM UTC.
The security patch will be applied gradually, starting with public rooms that have fewer members. A message will be sent in each room one day before the upgrade, as well as right before the process begins.
Why this is necessary
This is part of a coordinated response to security vulnerabilities that could potentially affect any server in the public federation.
The upgrade to room version 12 is the only way to ensure that our rooms are protected under the updated protocol.
You can read more details in the official Matrix announcement:
https://matrix.org/blog/2025/07/security-predisclosure/
We know this user experience isn’t ideal, but it is the only secure option available at this time.
Thanks for your patience and understanding while we keep our communications safe.
2 Likes
Hello,
Due to infrastructure issues, we are cancelling the room upgrades scheduled for tomorrow. We’ll announce a new date as soon as possible.
Thanks for your understanding.
1 Like
Hello,
Due to maintenance activities, we will be performing a reboot on one of the PostgreSQL database nodes. During this operation, there may be brief service impact, but the cluster will remain available.
The maintenance will start at 20:00 UTC.
We will share updates once the maintenance is complete.
Thank you for your understanding.
1 Like
Maintenance finished. 
3 Likes
Upcoming Public Room Upgrades
As part of the security patch announced by the Matrix.org Security Team, we will be upgrading all public rooms on our server to the newly released room version 12. This upgrade is necessary to address critical vulnerabilities in the federation protocol and ensure that our communications remain secure.
Below you can find the list and dates. Also, a message will be sent to the rooms before the upgrade.
1 Like
Hi,
The upgrade rooms task for today has been completed. The new IDs and future upgrades (once defined) are available in the post:
1 Like
Hello,
We got an incident on the synapse environment. This comes from some weird configuration on the underlying service automation that was using a wrong user. We did some cleanup which deleted that user which was in fact currently in use to manage the DB credentials. In parallel we got an incident with a physical host, that led to 2 Synapse workers being restarted and not able to access their DB.
It’s all fixed, since most Canonical is a Winter break, it took longer for resolution.
I’ll work with the team after the break to cleanup the situation as the resolution came from restoring the deleted user.
1 Like