Unexpected playbook behavior

This incident has been resolved. Please contact support if you see any remaining issues.

At approximately 1:30pm ET Thursday, March 6th, we shipped a bug to production that caused all “is not set” filters to evaluate to “false”. This caused a number of systemic issues:

- Lifecycle tracking rules may have been incorrectly applied.
- Many views and widgets were not showing accurate data.
- Playbooks with an “is not set” rule would have all of their enrollees unmatched and undone.

At 4pm ET, our Support team escalated widespread reports of these issues and an incident was declared. The team identified the pull request with the bug and reverted it. We also paused playbook execution pre-revert to evaluate impact. That revert finished at 5pm ET, at which point views & widgets started working again.

At that point, the team also forced a reprocess of lifecycle tracking rules, which corrected incorrectly tracked accounts and organizations.

However, the impact of re-enrolling entities into playbooks was potentially dangerous. So, the team started writing a restore job to reset playbooks to their prior place within the playbook run and re-do actions wherever possible. Writing and testing that job took until about 10pm ET Friday, March 7th. We have since run those jobs for all affected playbooks. As things stand now, we have reverted all undone playbooks to the best of our ability.

For playbooks with an “is not set” rule, enrollees would have seen the following action behavior:

- Assign a Key Role: *if* assignment strategy is “unassign”, would have unassigned affected enrollees. Those have been restored.
- Create an Indicator: would have ended the indicator. Those have been restored.
- Add to Segment: would have exited the segment. Those have been added back into the segment.
- Start a conversation, reply to a conversation: *if* undo strategy is “delete if unset” *and* messages had been created but not yet sent, they would have been deleted. This is unrecoverable.
- Create a task: *if* undo strategy is “delete the task”, tasks would have been deleted. Those have been restored.
- Update a task: *if* undo strategy is “revert the updates”, task updates would have been reverted. Those have been restored.
- Create a project: *if* undo strategy is “delete the project”, projects and incomplete tasks would have been deleted. Those have been restored.
- Create a doc: *if* undo strategy is “delete the doc”, docs would have been deleted. Those have been restored.
- Set a goal: *if* undo strategy is “delete the goal”, goals would have been deleted. Those have been restored.
- Update a trait: *if* undo strategy is “delete the trait’s value”, traits would have been set to null. Those values have been restored.
- Send a segment track: unaffected
- Ping a webhook: unaffected
- Show an NPS survey: NPS surveys would not have been shown to enrollees previously matching affected playbooks until they were restore.

However, there may have been second-order effects throughout the system that are difficult to identify. If one playbook was responsible for maintaining customer segmentation had an “is set” filter, it would have removed customers from segments. If a second playbook had filters on those segments, it may have also run incorrectly. All playbooks have since been re-run and should be reset to the most accurate state we’re able to attain right now.

If you still see systemic issues with any of your playbooks, please report them to our Support team via in-app chat or at support@vitally.io. We have a dedicated internal channel for these issues and will work to the best of our ability to minimize the impact on your business and customers. We deeply apologize for the impact this has caused you in managing your customer relationships and will continue to dedicate resources to ensure continuity of your business operations.

We are continuing to work towards recovery. At this point, we have restored data for most misapplied playbooks and are continuing to work towards resolving the remaining ones. Once we finish this portion of the incident response, we will be resuming all playbook execution. We will continue to do everything possible to expedite the resolution of this incident.

All playbooks remain paused while we work to correct the state of data to avoid propagation of unintended effects by partially restoring playbook execution. The team is working through this and apologizes for this disruption to your workflows

The incident has been isolated to playbooks that contain an “is not set” filter. Playbooks that do not contain an “is not set” filter have been resumed. Playbooks that contain an “is not set” filter remain paused.

Targets in playbooks with an “is not set” filter were incorrectly removed and may have had actions undone. We are working to restore those playbooks to the state they were before the bug shipped before resuming execution.

Due to a bug, we are experiencing unexpected playbook behavior. We have identified the root cause and are currently working to correct these unexpected behaviors. Playbooks are paused in the meantime.