Outage in Bonsai

Log4j Zero-Day RCE

Resolved Minor
December 10, 2021 - Started almost 4 years ago
Official incident page

Incident Report

The team is currently investigating the issue.

Need to monitor Bonsai outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Latest Updates ( sorted recent to last )
RESOLVED almost 4 years ago - at 12/11/2021 05:37AM

We have received independent confirmation via Elastic, Inc. that Elasticsearch is not vulnerable to RCE due to its use of the Java Security Manager. Our team will finish rolling out mitigations, but otherwise are standing down on updates here, pending any new developments.

IDENTIFIED almost 4 years ago - at 12/11/2021 12:59AM

All relevant versions for new cluster deployments have been updated, and we have re-enabled Sandbox cluster creation. We appreciate the patience from everyone who was stuck at the last step of new account creation this afternoon!

IDENTIFIED almost 4 years ago - at 12/11/2021 12:34AM

We are continuing to make steady progress in rolling out updates, with all of ES 5.x clusters updated, approximately 80% of ES 6.x, and over 50% of ES 7.x clusters updated.

IDENTIFIED almost 4 years ago - at 12/10/2021 09:17PM

Our team is continuing to roll out updates and making steady progress.

We’ve determined that a configuration based mitigation is not available in some early versions of Elasticsearch 5.x. Some customer clusters running on early versions of Elasticsearch 5.x have been upgraded to Elasticsearch 5.6.16.

Updates to Elasticsearch 6.x, 7.x, and OpenSearch 1.x are still under way.

IDENTIFIED almost 4 years ago - at 12/10/2021 06:17PM

At this time we're reasonably confident that Bonsai is not susceptible to the Remote Code Execution in this vulnerability.

However, we believe certain combinations of Java, Elasticsearch, and log4j can plausibly execute a remote ping. Out of an abundance of caution, we’re moving forward with a rollout of configuration mitigations.

For those following along and interested in the details of this incident, there are different combinations of the JDK version alongside the version of Log4j that are relevant to reproducibility. Per the security update from Apache (https://logging.apache.org/log4j/2.x/security.html)

>>>
Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against RCE by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".
<<<

Java 8u121 was released in January 2017, and Bonsai is running with newer versions of Java than that across the board. We believe this default has made our systems safe by default from this particular vulnerability.

IDENTIFIED almost 4 years ago - at 12/10/2021 05:40PM

We have temporarily disabled creation of Sandbox clusters pending updates to the underlying services.

IDENTIFIED almost 4 years ago - at 12/10/2021 05:18PM

Our engineers have identified the services within our platform which may be affected, however have not been able to reproduce the vulnerability. Out of an abundance of caution we are proceeding to roll out additional safeguards in the underlying service configurations.

INVESTIGATING almost 4 years ago - at 12/10/2021 05:18PM

We are continuing to investigate this issue.

INVESTIGATING almost 4 years ago - at 12/10/2021 05:11PM

The team is currently investigating the issue.

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook