MONITORING: Payment processing issues - [incident first identified 29 April/17:34 AEST in AEST/AEDT timezone])

This incident has been resolved. No further issues have been observed.

Summary:
PayID Alias Registration and Deregistrations exeperienced hightened errors between 29 April - 17:30 AEST to 18:09 and 18:15 to 18:29.
Due to the error rate only affecting a portion of traffic and Azupay redriving paymentRequest registrations, clients may not have experienced significant impact during the incident perioid.

While we initially identified a small count of hightened identified errors on outbound payment processing due to increased latency on an upstream API, these transactions were retried and did not have introduced delay in payment processing.

Root Cause of Issue
We have requested a root cause analysis from our Payment Accesss Gateway provider banking partner for additional information on the cause of the issue. Initial investigation and resolution indicates a degradation in a portion of their compute resources in their load balanced API responsible for PayID registrations, and cycling the unhealthy nodes resolved the issue. We will work with our banking partner to identify opportunities to improve the resiliency of these services.

We are monitoring the issue and system performance has recovered.

Our analysis shows the following impacts:

Elevated number of errors:
* Creation/Update of PayID's via api (Create PaymentRequest)

PayID Registrations have recovered post a second spike in errors between 18:10 and 18:28.
We have reprocessed paymentRequests that have been impacted.
Analysis of other API services indicate that no impact to other APIs has occured.


Our recommendations are as follows:

* Please inform your customers to re-attempt payments to paymentRequests if issues were experienced when attempting a payment.

Please do subscribe to our status page for the latest and most accurate information.

We apologise for the inconvenience caused and will provide additional information on root cause once available.

We are observing a recovery in impacted services as of 18:10 AEST and will continue to monitor to confirm resolution.

We are aware of issues in processing payments and payment request registrations. We are currently investigating the impact and will provide a detailed update as soon as one becomes available. The issue appears to be upstream of us affecting our banking partner and has been escalated to them for incident management.

Please do subscribe to our status page for the latest and most accurate information.

We apologise for the inconvenience caused.