Data acquisition and Ingest delay

We are now closing this incident. Thank you for your patience.

We have completed the hardware and network operations for this incident. We are keeping this in monitoring for 24 more hours to allow all changes over the last week to settle across the platform. At the end of this monitoring period we will be closing this incident as resolved. Thank you for your patience.

We made several changes last night and this morning to move traffic to new endpoints. We will have more maintenance windows in the coming days to move more traffic and increase throughput. Until then we will continue to monitor these changes. Most devices should now be up to date with minor delays to portal reporting.

We have made the switchover with our software and hardware updates and are monitoring the issue closely. We will update this message 3/12/24 9:00 AM PST.

We have completed the upgrade and cutover of our network devices. We see an improved infrastructure performance. The system is still processing the data backlog from the original issue, and we expect this to continue through the weekend. We will continue to evaluate the system through and decide on any other further action once the backlog clears.

We are continuing to make progress after doing additional testing and are actively upgrading specific infrastructure with our provider. That action requires swapping of equipment and other configuration information which has been going forward with an urgent level of support over the weekend between Stem and our provider. We will provide daily updates as we go forward.

STEM / Also Energy support and engineering teams have discovered network congestion in one of our data pipelines. This is causing delayed data ingestion for a subset of our fleet. We are working on steps to reduce congestion and increase bandwidth in the coming days. These steps should reduce network latency and will help with timely ingestion of data across our fleet.

STEM / AlsoEnergy support and engineering is diligently working on troubleshooting a delay in data ingestion affecting a subset of sites with Acquisuite loggers uploading data to our monitoring platform. We have identified the issue and are working closely with our 3rd party vendors to resolve it as soon as possible.

We are currently investigating this issue further.

We are continuing to monitor for any further issues after the scale-out undertaken late Friday evening

The underlying service contributing to latency is scheduled for scale out today, and we are continuing to monitor.

Backlogged devices are checking in while in night quiescence for generation. We are continuing to monitor latency reduction.

The issue has been identifed, a workaround implemented, with good initial results for device upload cadence.

We are seeing increase delay in some site data acquisition and ingest into the system . We are currently investigating the affected lagging sites.