Ravelin Status

Resolved Elevated BigTable Error Rate

There was an elevated error rate on the API causing a spike in 500s between 1934 and 1938 BST and another beginning now. This appears to correlate with a spike in BigTable CPU usage which we are investigating.

Resolved Increased Error Rate

We are currently investigating this issue.

Resolved Elevated error rates today at ~ 17:40 GMT

There was a transient problem with our main database around this time that put a lot of load onto a secondary database cluster that was under-provisioned. The problem corrected itself within 5-10 minutes, but we have increased the resources available to the secondary cluster in case the under-provisioning exacerbated the issue.

Resolved Elevated Error Rates

A continuation of the occasional burst of 500s being returned on our API: https://status.ravelin.com/incidents/pkr07psskgf2. Since our last update we have observed bursts of 500s returned from the API at a rate of 0.4% per minute at 2245 BST last night, and 0.55-0.76% per minute between 0545 and 055 BST this morning, and 0.035%. at 1130 BST. These 500s are the results of timeouts with our connection to Google BigTable, which we are continue to monitor. We are working with Google to investigate the matter further. In the mean time, we are adapting our internal request-routing logic to better identify which services are experiencing elevated error rates so that we can steer traffic to healthy services. We expect this to mitigate the impact of these timeouts while we investigate.

Resolved Elevated API Error Rate

We have observed several small bursts of 500s on api.ravelin.com and pci.ravelin.com today. The errors amount to roughly 2% of requests system-wide each time, and are the result of reads and writes to Google BigTable timing out. The impact of timeouts is inconsistent between hosts, and we are keeping an eye out for this issue re-occurring. This incident is a continuation of this morning's incident, which we continue to investigate: https://status.ravelin.com/incidents/xx5sm41vgr1y

Resolved Increased Error Rate

We are currently investigating this issue.

Resolved Dashboard unavailability

The Ravelin dashboard is currently unavailable. We are investigating to discover the cause. This outage affects the dashboard. Please check back here for further updates.

Resolved Loss of Fraud Score Services

We are currently investigating issues impacting our fraud scoring capabilities. Efforts are underway to restore availability. Updates will be provided promptly.

Resolved Partial degraded service

We are seeing elevated errors periodically on the API. Investigating...

Resolved Elevated Response Times and Timeouts

We are investigating increased response times and error rates beginning 19:30 BST. Initial indications are that these are coming from an increased error rate communicating with our primary storage, which had a spike in response times, but is beginning to return to normal as we process the backlog of requests queued for retry. We are investigating the initial cause of this spike now and continuing to monitor.