Ingest delays

This incident has been resolved.

Zookeeper corruption has been rooted out. Things appear healthier and catching up in all cases.

We are not to full resolution yet.

Throughput has improved although behavior of individual partitions remains a problem and is still causing delays in some cases.

It has been a long day with kafka. We continue to experience instability, causing lag and dropped payloads.

An alternate approach has been applied, we are watching.

The initial fix was unsuccessful, certain accounts are now substantially delayed.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.