Splunk AlwaysOn Memory Profiling Data Delayed

This incident has been resolved.

We are seeing a decrease in backlog as a result of the additional capacity added and expect resolution by 1:30pm PT.

At current rate, we anticipate resolution within 2.5 hours, we are looking for ways to accelerate recovery further.

We have not seen sufficient progress in the processing down of the backlog and are scaling up the system to address the issue, we will provide an update as it takes effect.

The delay in Memory Profiling data remains high, we are continuing to work on an estimate of when the backlog will be processed back down to acceptable levels.

We've determined the issue to be capacity related and have scaled up our systems to remediate it. We are monitoring the lag as it goes down. We will provide an update when we have an estimate on when we expect the incident to resolve.

Starting at approximately 5:39am PT a degradation in the performance of a backend component of Splunk APM is causing Splunk AlwaysOn Memory Profiling data to be delayed by more than five minutes. No data is being dropped at this time but data for the Memory Profiling page and Call Stacks on trace views are delayed.