Outage in BigPanda

Update on Sisense Customer Data Compromise

Minor
April 12, 2024 - Started 21 days ago
Official incident page

Need to monitor BigPanda outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including BigPanda, and never miss an outage again.
Start Free Trial

Outage Details

We want to inform you that we have identified and stopped some suspicious activities from an unidentified threat actor. Pursuing our audit of this security incident, we were able to identify there was read-only access to some user data. This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering. As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed. The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event. All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures. For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.
Components affected
BigPanda Third Party
Latest Updates ( sorted recent to last )
MONITORING 15 days ago - at 04/18/2024 05:20AM

We are continuing to monitor for any further issues.

MONITORING 15 days ago - at 04/18/2024 05:19AM

Following our latest notice on Saturday April 13th, 2024, we are writing to provide our Customers with an update on the latest available information regarding the Sisense Data Breach.

On Tuesday April 16, 2024, BigPanda was informed by Sisense that the Breach involved Sisense data cubes.

With this new information, we are finalizing a process to export the data that was compromised in the Sisense cubes and provide it to our Customers. We expect this process to be complete by the end of the week, at which time we will be able to schedule individual times upon request.

Our investigation is still ongoing. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.

MONITORING 20 days ago - at 04/13/2024 08:15PM

BigPanda became aware of a Data Breach through its subprocessor Sisense on Thursday April 11th, 2024. We informed Customers that same day of the Breach via the BigPanda Status Page (status.bigpanda.io). We are writing now to notify our Customers and provide an update on the latest available information.

Incident Summary
A Threat Actor compromised the Sisense platform and gained access to BigPanda Customer Data through Snowflake. To our knowledge, the situation has been contained. Our investigation indicates that at no time did the Threat Actor gain access to BigPanda’s Core Platform.

BigPanda’s Use of Sisense and Snowflake
The BigPanda Core Platform stores metrics data in Snowflake, which is then loaded into Sisense to deliver metrics reporting via Unified Analytics.

Incident and Data Scope
Sisense has confirmed the Breach but has yet to provide detailed information about the incident.

After BigPanda’s internal investigation, it was confirmed that the Threat Actor accessed BigPanda Customer Data stored in Snowflake via Sisense. It was determined that a read-only service account credential for Snowflake was compromised due to the Breach, thus enabling the threat actor to make SQL queries on the Snowflake database.

The BigPanda Customer Data accessed included usernames (which contain BigPanda customer email addresses) and alert and incident tag information. This data in Snowflake enables customers to identify trends within their environments, examples are included below:
Username (email address), app_id, source_host, priority, start_time, end_time, date, etc

Incident Containment & Remediation
Upon becoming aware of the incident, the BigPanda Security Incident Response Team immediately launched an investigation, which included execution of the following steps:
- Requesting information and contacting Sisense;
- Reviewing all internal systems audit logs for abnormal activity;
- Performing programmatic updates to rotate user access keys and BigPanda’s SSO token via prepared scripts;
- Rotating service account keys for systems that feed data to Sisense cubes.
- Stopping any new Customer Data from being sent to Sisense. We will re-evaluate this decision in the days to come based on information that will be provided to us from Sisense.

A more detailed list of the actions taken by the BigPanda Security Incident Response team to ensure the containment of the Breach are as follows:

- Changed all Sisense-related passwords on my.sisense.com
- For non-Single Sign-On (SSO) access:
-- Replaced the Secret in the Base Configuration Security section with your GUID/UUID.
-- Reset passwords for all users in the Sisense application.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
- For SSO access:
-- Updated sso.shared_secret in Sisense and updated the newly generated value in the SSO handler.
-- Rotated the x.509 certificate in our SSO SAML identity provider.
-- Rotated the OpenID client secret.
-- Updated SSO settings in Sisense with the revised values.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
-- Reset credentials in the database used by the Sisense application.

Ongoing Investigation and Mitigation
BigPanda is continuing to investigate the issue and will conduct a full external forensic investigation of its Data Platform. We will continue to provide updates as they become available.

For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.

MONITORING 21 days ago - at 04/12/2024 07:08AM

We want to inform you that we have identified and stopped some suspicious activities from an unidentified threat actor. Pursuing our audit of this security incident, we were able to identify there was read-only access to some user data.

This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering.

As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed.

The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event.

All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures.

For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.

Latest BigPanda outages

Search Latency - 23 days ago
Pipeline Latencies - about 1 month ago
Pipeline Latencies - 4 months ago

The easiest way to monitor BigPanda and all cloud vendors

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3154 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime