Load balancer outage

Access to applications:
All end-user's applications have been applied their respective SSL certificate. Access to application is fully operational now

Schedule Reports:
All queued reports have already been sent

Access to applications:
We have generated almost all SSL certificates to ensure access to end-user's applications are labeled as a secured connection. This will be ready today.

CDN static files:
Some users will experience loss of some white-label assets (read previous update).
Unfortunately, we cannot recover them, so the fastest way to get them back into the platform is simply upload them again.

Schedule Reports:
There's still some user reports still repressed in the queue. This process is DB intensive because of the amount of data that needs to be extracted. We will continue to update as the queue clears.

Access to industrial.ubidots.com has been completely restored.

Some end-user application are still missing its SSL certificate to ensure a secure connection, so our DevOps team is head on making sure this is applied to all applications as soon as possible.

The hard drive failure corrupted static files hosted in the CDN, including assets such as those found in the applications' white-label (Loader logo, Favicon, Header logo, Login screen logo, Background image) configuration, as well as for example, the Profile's avatar image.
Some user might have to upload this files again.

We have also identified that the Scheduled Reports engine was also compromised because of this hard drive failure. This means that as of this moment, there's a queue of reports to be generated and sent. Some of them might come up with errors.

Access to the main admin accounts at industrial.ubidots.com has been almost completely recovered as we have restored the CDN from which static files are loaded. Some users might experience degraded performance still.

End-user's applications static files backup has been mounted and we are generating the SSL certificates for the applications. As they're generated, they are being applied, which means that some applications will start working while others will take more time.
Our DevOps team is estimating one more hour for all certificates to generate.

We are in direct contact with our infrastructure provider to find a solution to the disk failure.

At this moment, access to the main accounts at industrial.ubidots.com is intermittent, while the end-user's applications are still not accesible. We're mounting a backup of the application's static files to speed up the process of getting the UI back into operation while we have a more definitive solution to the disk failure from our provider.

We have identified the error to be coming from a hard drive failure in the load balance and CDN server.
We have moved all data traffic to a new load balancer to stop it from being lost, however, the UI is still not loading, and we're working on mounting a backup of the CDN.

As of this moment we know there will be a gap of data from 23:00 to 23:30 UTC.

We are continuing to investigate this issue.

We have experienced an error in one of our servers, specifically the one containing our load balancer for data traffic and the CDN from which static files are served.
This is preventing data from being ingested through any of the supported protocols (HTTP, MQTT, TCP/UDP). Also, it is affecting the UI from loading, both at industrial.ubidots.com and the end-user's applications

We're investigating the root cause.