Temporarily suspending upgrades to Aiven for PostgreSQL version 16 due to bug in upstream PostgreSQL

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). A root cause has been identified and our engineers are working on a workaround to enable upgrading existing Aiven for PostgreSQL services to version 16. In addition, we are also working together with the PG community to propose a proper fix for upstream.

We are confident that the bug can't lead to corruption of customer's data or indexes on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

Current Status:

• The creation of new Aiven for PostgreSQL 16 services is available again.

• Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

• We have identified repair actions where this issue occurs.

• We have found the root cause and we are working on the fix.

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). We are still focused on finding the root cause and continuing on creating a fix.

We did not find any evidence that the bug has led to corruption on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

This scenario appears more frequently on small instances and is only affecting a small number of Aiven for PostgreSQL 16 services.

Current Status:

• Creation of new Aiven for PostgreSQL 16 services is available again.

• Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

• We have identified repair actions where this issue occurs.

• We are continuously focusing on finding and fixing the root cause of the issue.

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). We are still focused on finding the root cause and creating a fix.

We did not find any evidence that the bug has led to corruption on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

This scenario appears more frequently on small instances and is only affecting a small number of Aiven for PostgreSQL 16 services.

Current Status:

• Creation of new Aiven for PostgreSQL 16 services is available again.

• Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

• We have identified repair actions where this issue occurs.

• We are focusing on finding and fixing the root cause of the issue.

Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service.

Current Status:

• Creation of new Aiven for PostgreSQL 16 services is available again.

• Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service.

Creation of new Aiven for PostgreSQL 16 services is available again.

Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service. Shortly, creation of new Aiven for PostgreSQL 16 services will become available again.

Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

We continue to disallow the creation of new Aiven for PostgreSQL services using PostgreSQL version 16. This is due to an ongoing investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

Please note that the PostgreSQL versions 12 to 15 are still available for creation.

We apologise for the inconvenience caused by this issue as we continue our investigation and resolution.

We will provide another update in the next few hours.

We continue to disallow the creation of new Aiven for PostgreSQL services using PostgreSQL version 16 due to an investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

We apologise for the inconvenience caused by this issue as we continue our investigation and resolution.

We will provide another update in the next few hours.

We have temporarily prevented the creation of new Aiven for PostgreSQL services using PostgreSQL version 16 due to an investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

We apologise for the inconvenience caused by this issue.