Load API Integration Downtime

Issue Summary Due to a major release on 7/12/2022, we introduced a breaking change that caused all Load API integrations to intermittently fail. This issue affected all customers on API integrations, which includes many of our direct TMS Integrations. These integrations include: - Aljex - McLeod - Tai - Turvo - Revenova - FMS TMS - EZLoader TMS - 3PL Systems - Any Homegrown TMS's that are API integrated CSV integrated customers were not affected. Timeline The issue was identified at 6:00AM PDT. The escalation and fix identification was done by our engineering team promptly, and the eventual resolution was introduced into our production environment at 8:25 AM PDT. Root Cause The cause of API integrations to fail was because of a new data field we had introduced for a specific TMS integration. This field was intended to be a completely optional new field, with no backwards breaking changed. However, the code deployed in the release made the field required, which broke many existing API integrations. Resolution and recovery The cause of the downtime was quickly identified because of our clear release logging. The eventual fix needed a code change, which was introduced into our staging environment for testing, before it was eventually moved into our production environment at 8:25 AM PDT. We have daily syncs for most API integrated customers/TMS's to make sure we are in a proper load state by the end of the day. Some API customers may need to trigger new updates on their loads from the TMS end to recover from any missing updates. Customer Success will reach out to customers on a case by case basis if additional updates are needed from the TMS. Corrective and Preventative Measures Main cause of this issue was the lack of QA for the integration related change that caused the backwards incompatibility. We are making changes to our QA process and our staging environment monitoring to make sure that any critical integrations changes are properly tested. We are also building in regression tests to make sure we have no backwards incompatible changes. Some of these improvements have already been implemented, but we expect full testing and QA pipeline improvements to come in the next few weeks.