Axiom and Accounting Axiom Degraded Performance

We have been carefully monitoring the system today and it has remained stable with the improvements made this week, and are therefore marking this issue as resolved.

The issue stemmed from a problem in loading Workspaces for users, which show up in a few places (such as the Workspaces tab, search results, and when saving a query). Because this is a frequent operation in the system, the issue had the side effect of impacting general Axiom performance. We disabled this feature in order to keep the rest of the system available, deployed several improvements, and then re-enabled it, resulting in a stable experience.

We have rolled out an update to the part of our back-end system that we identified as the cause of the issues. These updates have yielded noticeable performance improvements, and as a result we have restored all functionality. The system continues to be stable after restoring all functionality, but we will be monitoring performance closely tonight and tomorrow to confirm the issue is fully resolved.

We continue to work on restoring Workspace related functionality as our top priority. We have restored the ability to delete and overwrite queries and charts; however, saving new queries or charts is still disabled. Workspaces pinned to homepages have been partially restored; clicking on them will show their queries but not nested workspaces. Finally, navigating to Workspaces is still disabled. This is our current focus. The next update will be no later than 5pm Eastern (22:00 UTC).

We have identified the problematic area of the system, but have not yet identified a solution. This continues to be our top priority, but at this time we do not yet have an ETA for resolution. The next update will be provided at noon ET (17:00 UTC) tomorrow.

Axiom Workspace and query saving restrictions continue to be in place. Bringing these back to normal is our top priority, though we do not have an ETA for resolution yet. Another update will be provided by 5:00pm Eastern. If you need help finding queries that are saved in your workspaces, please open a ticket and we will be happy to assist you.

Axiom performance is stable following the emergency maintenance and temporary workspace and query restrictions that have been put in place. These temporary restrictions will remain in place while we work toward a full resolution for this issue. Please refer to the email communication sent to school point people for more details about this incident.

We are performing emergency maintenance at 4:30 PM ET (21:30 UTC) to help address the underlying issue impacting Axiom performance. This will result in a brief period of downtime (approximately 15 minutes) for applications impacted by this issue: Axiom, Data Export Packages, and the Data API.

We are aware of continued performance issues and are actively working toward a resolution.

In order to restore Axiom as soon as possible, we have temporarily made workspaces unavailable. Workspaces will not show in the workspace picker or on homepages.
Additionally new queries are unable to be saved, and existing queries are available but cannot be updated or deleted at this time.
We are continuing work to restore full functionality for workspaces and queries, and will provide further updates as soon as possible.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.