BLiP Platform [Degradation]

Since the last communication, we have not identified any monitoring instability, so we are closing the incident.

The indicators show normalization of the environment, so we are moving the state to monitoring.

Actions in progress
- We are in the process of finalizing the environment scaling.
- The scenario will continue to be analyzed and definitive correction actions will be shared in the technical report of unavailability.

Status: Crisis room closed, we continue to monitor

After the restart actions, the impact for customers is mitigated

Actions taken so far:
- Restart of services to mitigate the scenario.
- Scale the environment to improve platform performance with completion forecast at 17:15.

Actions in progress:
- Analyzing and planning improvements for definitive corrections of the problem.

Next update: in 30 min or when a relevant new fact is presented;

Scenario: Dropped connections in some services impacting the use of the platform

Actions taken so far:
- We reopened the crisis room
- Restart of services to mitigate the scenario

Next update: in 15 minutes or when a relevant new fact is presented;

We are continuing to investigate this issue.

We detected a drop in our services, presenting a scenario similar to today's morning period, we are investigating.
Actions in progress:
- Scale the environment to contain degradation

Actions in progress:
- Scale resource ended, monitoring shows reestablishment of normal behavior.

Next update: final communication, the monitoring status will be kept for a period, when we will make a final communication.

Actions in progress:
- Started the resource scale process that is impacting the services;
- Some services keep restarting to try to mitigate the errors.

Next update: in 15 minutes or when a relevant new fact is presented.

After restarting the applications and a brief moment of improvement, the problems reoccurred, some services may have been impacted due to their high resource consumption, and some are still stopped on some machines.

Actions in progress:
. Deployment of the new release is in progress;
. We will start the process of scaling the resource that is impacting the services.

Next update: in 15 min or when a relevant new fact is presented.

After restarting the platform applications, we observed an improvement but then a new drop. We are seeing errors in the media application that may have been caused by high consumption of some resources.

Action in progress:
Deploying a new version of the currently impacted service.

Next update: in 15 minutes or when a relevant new fact is presented;

- Monitoring shows that services are starting to respond after being restarted;
- Time continues to investigate and restart the platform applications to mitigate the problem.

Next update: in 15 minutes or when a relevant new fact is presented.

Affected services are being restarted for platform normalization.

Next update: in 15 minutes or when a relevant new fact is presented.

We are experiencing from an partial outage on Blip Platform

Impact:

Fail on exchanging messages with bots on all channels. Platform unavailable to attendants

Update:

Our technical team is already working to resolve the issue.