Placements platform experiencing degraded service due to third party infrastructure provider issue

This incident has been resolved.

Placements is fully operational at the moment - our third party infrastructure provider has rolled out a fix.

The platform has caught up on most backend tasks and CRM and ad server imports should be functioning normally again.

We have recovered some ability to scale up workers manually and are now able to process billing reports. CRM imports are now processing faster as well after scaling up workers.

We have received the following update from our infrastructure provider:

"We’ve identified the likely trigger of the issue and are making strong progress toward a resolution. A recent automated update disrupted network connectivity on affected instances. We're working on a fix to prevent this from recurring and will provide an ETA for resolution when it's available."

We received the following update from our infrastructure provider:

"We continue to work on the permanent fix and explore alternate paths to resolution. Once the fix is developed, an estimated time to resolution will be shared."

Our third party service provider continues to investigate and remediate the issue.

Billing reports and hour imports, along with other backend jobs (like email notifications) continue to be backed up.

We are continuing to investigate this issue.

An ongoing incident with an infrastructure provider is causing degraded services on the platform. Specifically, we are unable to initiate scheduled jobs or spin up new machines to process queues.

The app generally functions fine but scheduled ad server and CRM imports are paused and scheduled billing jobs can't be run.

We are monitoring the incident with our infrastructure provider.