0 incidents in the last 7 days
1 incidents in the last 30 days
Last check: 3 minutes ago
Last known issue: 30 days ago
Resolved Vero UI unavailable
We are have been experiencing intermittent issues causing parts of Vero's UI to be slow or unresponsive, we are currently working to resolve the issue.
Resolved Messaging delays
On Tuesday 23rd March at approximately 6:30pm UTC we observed delays in processing automated messages (behavioural, transactional and workflows). We identified an issue with our round-robin system. We debugged and fixed the root cause by approximately 9.30pm UTC. During the period 6:30pm-9.30PM UTC automated messages were operating at a severely reduced throughput, resulting in a backlog of automated messages. The backlog of automated messages was fully processed and running in realtime by 11pm UTC. Newsletters sent between 9.30pm-12:30am UTC (Wednesday) were sent at a reduced rate whilst this backlog was processed, resulting in some delays. We've been posting updates to the various system components on this status page over the last 12 hours to reflect the above delays as they occurred. -- Since 12:30am UTC we have identified a second issue with some of the raw hardware provided by our cloud provider. **It appears this is a hardware issue and is not related to the earlier incident**. Messages **are** currently sending in realtime but we are operating at an overall reduced capacity with some delays in aggregating conversion metrics and sending webhooks. We're working with our cloud provider to resolve this issue, at which time we'll be able to scale to full capacity. In the interim we're monitoring and will post updates if the situation changes and message send speeds are affected. If you have any questions, please email us at [email protected]
Resolved Elevated API Errors
We're experiencing an elevated level of API errors and are currently looking into the issue.
Resolved Issues calculating segments and sending newsletters
We are currently investigating an issue affecting the calculation of segments which will mean your newsletter may experience delays if attempting to send.
Resolved Email template discrepancies
We're investigating reports that some email templates are rendering differently than expected. If this is the case you should see these emails being discarded instead of sent. This ensure's that users will not receive incorrect content. Please email [email protected] if you have any questions regarding your account or emails.
Resolved API connections issues
We have been alerted that there are reports of connection issues to our API and we are investigating.
Resolved Delays in email processing and statistics
On Wednesday 27th May at approximately 10am UTC we observed intermittent delays in API processing and, from approximately 12pm, delays in workflow and behavioural email processing. These delays were not fully resoled until 4pm UTC. From that time, the queues were processing normally. We have made several changes in response to the delays seen on the 26th and 27th May and are confident that delays in email or API processing should not occur again. We monitored these queues through Thursday 27 May and will continue to do so. Both API and email status components were updated to reflect the delays as they occurred. If you have any questions, please email us at [email protected]
Resolved Delays in email processing and statistics
We are sharing this post mortem to provide an easy-to-navigate record of our prior component updates this week. On Tuesday 26th May at around 12pm UTC we began observing intermittent delays in email processing and statistics collection. Upon investigation, this appeared to be related to some infrastructure changes we had recently deployed. Our Operations Team was able to manage many of these delays as they occurred, but message queues for **some** customers were delayed on and off between 12pm UTC and 6pm UTC. From 6pm processing had returned to real time. In response to these delays, we reviewed and adjusted the previously deployed infrastructure changes. We have updated the status of all relevant components in real time throughout the week. If you have any questions, please email us at [email protected]
Resolved Increased API error rates
At 17:18 UTC 18 May 2020, one of the databases backing our API began experiencing issues. Between 17:18 and 17:21 a small number of API requests were dropped (approximately 800 total). We responded to the issue and between 17:21 and 17:40 we were able to rectify the core problem. By 17:47 we were processing at normal capacity. It took approximately one hour to work through the backlog of API requests that had accumulated. Triggered messages were delayed during this period. No data was lost. Aside from the ~800 dropped requests, no data or emails were lost. This issue was caused by a regression introduced with a recent infrastructure change. We are reviewing our tests and QA moving forward. Thanks. If you have any questions, please write to us at [email protected] — we're happy to answer them.
Resolved Delayed email processing
At 17:32 UTC yesterday (29th April) we observed degrading performance in processing emails and updated the email status component whilst investigating. We identified an increasing number of errors related to an associated database, which we modified and rebooted. We were able to quickly regain access to the affected database and begin processing the backlog of emails by 18:20 UTC. Email processing and statistics were delayed for a total 48 minutes and are now processing real time. All API processing and segment queries remained realtime throughout and were not affected. If you have any questions about your account please contact [email protected]
We're monitoring 806 services and adding more every week.
IsDown integrates with hundreds of services. Handles the hassle of going to each one of the status pages and manage it one by one.
We also help control how you receive the notifications.
We monitor all problems and outages and keep you posted on their current status in almost real-time.
You can easily get notifications in your email, slack, or use Webhooks and Zapier to introduce the service status in your workflows.
IsDown collects status data from services to help you be ahead of the game.