Flexera One – SaaS Manager – NA – Integration Tasks Failing Intermittently

This incident has been resolved.

We will continue to monitor the environment over the weekend. We will also keep this incident open till Monday morning as long as no further issues are detected.

Health checks have been successful and technical teams have not observed any errors or issues in the logs since the change was deployed to production yesterday. We will continue to monitor the environment till the EOD tomorrow.

The fix has been deployed to Production. We are currently monitoring the change.

The fix is ready to be deployed to the non-production environment today. We will be monitoring the change over the next day few days before deploying it to production.

Teams are actively working on a code change which will be later deployed to Staging for further testing. Due to the intermittent nature of the issue, the plan is to let it run for a few days in staging to ensure we do not encounter the same errors before production deployment; and perform additional testing as needed.

Meanwhile, as a temporary workaround, we will continue to manually re-run the failed tasks to remediate any customer impact.

We are continuing to actively work on a fix for this issue.

The investigation has revealed that there have been only a few occurrences since Saturday, October 22nd, impacting a very small customer base. We have also identified the cause of the failures and are currently working on a fix. We are continuing to monitor the environment and rectify any problems requiring manual intervention.

Around 1:37 AM PDT today, integration tasks failed again with 'signing failed "500 Internal Server Error". The outage may have lasted for approximately 10 mins. Services have been stable since, and teams are investigating the error logs.

We have not observed any issues and tasks have been running successfully since the additional resources were deployed. Technical teams will continue to monitor the tasks for another day to ensure a successful run.

We are continuing to monitor for any further issues.

We have added additional resources in the environment to avoid any memory contention issues. Technical teams are monitoring further.

There have been no reoccurrences of this incident since Sunday, October 23rd. Integration tasks have been running successfully without errors. However, the root cause is still unidentified. We are continuing to investigate the root cause to avoid any reoccurrences.

Incident Description: Customers using SaaS Manager may receive intermittent errors while trying to retrieve daily Integration tasks data via Managed Applications. These tasks run only once every day. If the task gets stopped by the error one day, it will attempt again on its regular cadence the following day. As a result, the availability of new information could be delayed by a day.

Priority: 2

Restoration activity:
Technical teams have been engaged and are currently investigating