Possible Platform Instability This Morning

Overnight, Heroku have announced that all further issues on their end have been resolved. We've seen no further impact from this incident overnight, and are therefore declaring it resolved.

At present, Zipline is seeing little to no impact to our services from the ongoing daylong disruption to Salesforce/Heroku systems.

Salesforce are continuing to work to stabilize their systems, and are currently initiating a fleetwide reboot of all machines across their Cloud services, including Heroku. It is possible that this may cause one or more brief service outages for Zipline overnight tonight, depending on if/when Salesforce reboot the clusters which house our application.

Our on-call team remains ready to respond to any impacts overnight, and our Platform engineers will reassess the current state of play first thing tomorrow morning.

Thank you for your patience as we weather someone else's storm together.

Zipline systems continue to be minimally affected by Salesforce's troubles, and we have further altered our resource deployment so that we can better weather any continued issues overnight and into tomorrow.

We continue to lack a firm resolution timeline from Salesforce. Their most recent update is as follows:

"We don’t have additional information to share at this time. Our efforts remain focused on internal testing and validation as we continue to see incremental improvements. We will provide a resolution timeline as soon as possible, and we apologize for the continued trouble."

We continue to see negligible impact to our services at this time, but Heroku have not yet indicated that they have fully resolved issues in their system, so we are continuing to monitor the situation closely.

Salesforce/Heroku's most recent update, as of 12:46pm PDT, was as follows:

"We are observing positive results in our initial testing on pre-production environments. We’re continuing to test the fix before applying it to production environments and will provide a resolution timeline once determined.

We will provide another update within 30 minutes or sooner if new information becomes available."

We are seeing very little impact to our services currently, but Zipline's team will remain in an active alert state until we hear that a definitive fix has been deployed by Heroku/Salesforce.

Salesforce are continuing to work on a permanent fix for their issues today. In their words:

"We continue to work on the permanent fix and explore alternate paths to resolution. Once the fix is developed, an estimated time to resolution will be shared."

We will pass on Salesforce's resolution ETA as soon as they state one.

We are confident that we're now able to absorb the worst of the impacts from Heroku's ongoing instability, but cannot guarantee uninterrupted service until Salesforce/Heroku assure us that they've completed work on their systems.

Salesforce's most recent update, as of 9:38am PDT, is as follows:

"Investigations are still ongoing. A permanent fix is in progress, and it will require disabling the configuration change that reintroduces impact."

We are currently seeing a significant improvement in the responsiveness of our underlying Heroku services, and the stability of our applications as a result.

Our team now also has at least partial access to the tools we need to manage our system resources effectively.

We will continue to monitor this as an active incident until Salesforce/Heroku give a definitive "all clear" on their systems.

It appears that Salesforce have identified the root cause of the issue, but have yet to fully resolve it. From their status updates (https://status.salesforce.com/generalmessages/10001540):

"Some customers are seeing improvements while accessing their services; however, the current impact still remains. Further investigations highlighted that an automated change on the Heroku environment was the potential trigger of this issue."

Within the last 20 minutes, Salesforce has began rebooting affected Heroku instances and claim that this should restore service.

We are seeing error rates drop across our hosting infrastructure, and will continue monitoring closely until we are confident that service is fully normalized.

We are continuing to monitor the situation closely with the utmost urgency as our top priority at this time. To recap, Zipline is currently impacted by a major upstream outage being experienced by our hosting provider Heroku (A Salesforce Company). More details on that can be found here: https://status.salesforce.com/generalmessages/10001540

Depending on the Zipline cluster where your organization is being hosted (i.e. NA, EU, etc.) you may be having a different experience simply based on the differences in overall traffic/volume/load. Generally speaking though, attempts to reload, refresh or log out and back in to the application will not serve useful.

Our teams will continue to remain close to this issue and provide updates via this channel as soon as they are available. We apologize for any inconveniences or disruptions this may be causing to your day to day operations.

We are continuing to monitor for any further issues.

We are continuing to monitor the situation closely with the utmost urgency as our top priority at this time. To recap, Zipline is currently impacted by a major upstream outage being experienced by our hosting provider Heroku (A Salesforce Company). More details on that can be found here: https://status.salesforce.com/generalmessages/10001540

Depending on the Zipline cluster where your organization is being hosted (i.e. NA, EU, etc.) you may be having a different experience simply based on the differences in overall traffic/volume/load. Generally speaking though, attempts to reload, refresh or log out and back in to the application will not serve useful.

Our teams will continue to remain close to this issue and provide updates via this channel as soon as they are available. We apologize for any inconveniences or disruptions this may be causing to your day to day operations.

Our hosting provider, Heroku, is currently investigating an issue in their platform, which hosts the Zipline app:

https://status.salesforce.com/generalmessages/10001540

https://status.heroku.com/incidents/2822

As a result of this upstream issue, our engineers currently do not have access to many of the tools that allow us to manage system load, and we understand that the systems which "autoscale" new resources to handle busy peaks may also be offline.

While this issue is ongoing, it's possible that our service will be degraded as it becomes busier over the course of the day.

We will continue to directly monitor Heroku's (so far scant) information regarding the causes and likely resolution time for this incident.