Aptible Status

Resolved Increased Error Rate

We are observing an increased error rate during some operations for App and Database operations due to an increased AWS API error rates. AWS has reported that they are aware of the issue (https://status.aws.amazon.com/).

Resolved Deploy API performance issues

We are currently investigating long response times from the Aptible Deploy API, which may result in performance issues browsing dashboard.aptible.com or operations that fail due to timeout.

Resolved Degraded website performance

Our authentication service has seen a sudden increase in response times. This is causing degraded performance on our services as they communicate with the authentication service. We are currently investigating to determine a root cause.

Resolved EC2 Host Failure

We are investigating an EC2 dedicated host failure affecting a small number of apps and databases in us-west-2. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).

Resolved EC2 Host Failure

We are investigating an EC2 dedicated host failure affecting a small number of apps and databases in eu-west-2 region. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).

Resolved CVE-2020-14386 Patching

Recently a Linux kernel vulnerability (CVE-2020-14386), which allows for memory corruption and possible privilege escalation, was announced [0]. Our security team began investigating and preparing our remediation strategy yesterday (September 3) at 22:00 UTC. As part of our remediation efforts, we will be restarting all App, Bastion, and Build instances in shared stacks today. We will update this status page if any further action is required. [0] https://seclists.org/oss-sec/2020/q3/146

Resolved Updating shared environments with latest speculative execution attack mitigations

To protect our customers’ shared environments against speculative execution and branch prediction vulnerabilities (Spectre), our Reliability Team will be re-launching customer containers and utility services (e.g., build instances, ephemeral SSH servers) in the following in shared environments between 10am and 6pm on Tuesday, September 1st. • App instances • Build instances (where Docker images are built) • Bastion instances (where ephemeral `aptible ssh` containers run) The new EC2 instances have been updated to the latest Linux kernel, which contains additional mitigations and fixes against some variants of these attacks. Instances in shared environments where arbitrary code may be run are the most vulnerable and have been prioritized for replacement. The replacement operation should incur no downtime for your apps, but will require that we run `aptible restart` (a zero downtime operation) on each app in your shared environments. There will be no impact on dedicated stacks during this maintenance. We will continue to update this status page as we proceed with the replacement process on Tuesday.

Resolved EC2 Host Failure

We are investigating an EC2 dedicated host failure affecting a small number of apps and databases. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).

Resolved SSL Configuration Error for Certain Databases

At 13:40 UTC today, our Reliability Team identified an issue affecting the SSL configuration for certain Aptible Deploy databases. Specifically, we identified an issue with the certificate bundle provided by our SSL vendor, Namecheap, for the `*.aptible.in` certificate we use to secure some databases via SSL. Despite the fact that the certificate itself expires in February 2021, Namecheap provided us with a certificate bundle whose root certificate expired on May 30, 2020. We were not aware of this error until today, and Namecheap did not notify us about it. Most database client libraries will still successfully connect to these databases even though the root certificate is expired, since the primary and issuer certificate are both valid. However, out of an abundance of caution, we are restarting all databases of the following types, to avoid issues: * Elasticsearch * InfluxDB * CouchDB * RethinkDB In the unlikely event you see SSL-related issues raised by your app in connecting to databases of the following types, please simply restart your database using `aptible db:reload DATABASE_NAME`. This will automatically restart your database with the latest certificate (and bundle). Because of the very low probability that clients of these databases would be affected, we are choosing not to proactively restart them, as that might cause a brief disruption of service to customers. * MongoDB * Redis * RabbitMQ We will continue to update this issue as the database restarts mentioned above are completed, and if there are any changes to our planned remediation.

Resolved EC2 Host Failure

We are investigating an EC2 dedicated host failure affecting a small number of apps and databases. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).