Outage in CardConnect

BluePay Transaction Data Partially Unavailable

Minor
June 19, 2024 - Started 5 months ago
Official incident page

Need to monitor CardConnect outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including CardConnect, and never miss an outage again.
Start Free Trial

Outage Details

A connectivity issue between BluePay database servers caused transaction data from transactions processed 6/18/2024 4:16pm-4:26pm EDT to not be merged to all database servers. Affected transactions are currently not available in the BluePay website and not available from reporting APIs. We are currently working to correct the issue. Once connectivity is restored between the database servers, the transaction data will be merged making it available in the BluePay website and reporting APIs. The transactions will then settle in the next regularly scheduled settlement.
Components affected
CardConnect BluePay Gateway
Latest Updates ( sorted recent to last )
IDENTIFIED 4 months ago - at 08/07/2024 10:18PM

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 were submitted for funding on 8/5, merchants should see funds processing through their accounts.

Please continue to monitor account activity.

We will continue to provide updates as they become available, if there are questions or concerns please contact your support team.

IDENTIFIED 4 months ago - at 08/07/2024 04:15PM

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 have been reviewed and prepared for settlement. Funding should be expected within the next 48 hours.

Please continue to monitor account activity.

We will continue to provide updates as they become available.

IDENTIFIED 4 months ago - at 07/26/2024 04:16PM

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 are in the process of being manually closed once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/10/2024 04:43PM

Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items.

We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/09/2024 04:15PM

Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items.

We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/08/2024 02:09PM

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed.

Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/03/2024 05:01PM

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed.

Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/02/2024 04:43PM

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are being worked on.

Invalid batch sequence errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/02/2024 01:55PM

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are being worked on.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/01/2024 05:57PM

Duplicated settlement reversals for all ACH transactions have been submitted.

North and Omaha duplicated settlement reversals have also been submitted.

We are continuing work through recovery efforts including correcting transaction records.

A full impact list will be provided to account managers to be provided upon request in the coming days.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 07/01/2024 03:32PM

We are continuing to correct the remaining double settled transactions.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 06/27/2024 03:02PM

We are continuing to correct the remaining double settled transactions.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 06/26/2024 02:12PM

Double settled Fiserv ACH Third Party Sender transactions have been reversed. We continue to work on reversals for Fiserv ACH Direct to Bank and credit card transactions.

We will continue to provide updates as they become available.

IDENTIFIED 5 months ago - at 06/25/2024 05:51PM

Duplicate settlements on the BluePay Gateway are currently in the process of being reversed. This may take up to 3 business days to be completed.

Please subscribe to status.bluepay.com for updates to the timeline as we have them.

IDENTIFIED 5 months ago - at 06/25/2024 12:45PM

We are continuing to work on a fix for this issue.

IDENTIFIED 5 months ago - at 06/24/2024 09:52PM

We are continuing to work on a fix for this issue.

IDENTIFIED 5 months ago - at 06/23/2024 08:36PM

Reversals of double settled credit card transactions over $10,000 on First Data processing platforms are pending settlement. Transaction Type is CREDIT for reversal of a SALE or CAPTURE. Transaction Type is SALE for the reversal of a REFUND. The Comment on the transactions is "Reversal of double settlement.".

Reversal of double settled credit card transactions under $10,000 on First Data processing platforms is starting now. It is expected to be completed Monday 6/24/2024.

IDENTIFIED 5 months ago - at 06/23/2024 06:31PM

Reversal of double settled credit card transactions over $10,000 on First Data processing platforms has begun. Reversal of smaller First Data credit card transactions, credit card transactions on other platforms and ACH transactions are still being planned.

IDENTIFIED 5 months ago - at 06/23/2024 06:30PM

We are continuing to work on a fix for this issue.

IDENTIFIED 5 months ago - at 06/21/2024 08:11PM

Transactions that previously received a "COULD NOT LOCATE ORIGINAL TRANSACTION" error can now be reprocessed. The original transactions have been restored.

IDENTIFIED 5 months ago - at 06/21/2024 04:32PM

We are continuing to work on a fix for this issue.

IDENTIFIED 5 months ago - at 06/21/2024 03:17PM

During this time please refrain from manually refunding transactions appearing as duplicates. This will be resolved once a fix is in place.

IDENTIFIED 5 months ago - at 06/21/2024 12:41PM

We are continuing to work on a fix for this issue.

IDENTIFIED 5 months ago - at 06/20/2024 08:18PM

The data synchronization issue has caused some transactions to be double settled. After the missing data is restored we will be identifying the double settled transactions and determining the best way to reverse the duplicates.

IDENTIFIED 5 months ago - at 06/20/2024 06:01PM

An application to synchronize the data across the database servers has been created. We are starting the process to restore the missing data. A completion time is not known yet.

IDENTIFIED 5 months ago - at 06/20/2024 01:36PM

A database server added to the cluster is failing to synchronize with the other database servers. This is causing some transaction and settlement data near the time of the addition to not be available for reporting and for use as tokens. We are working on correcting the synchronization issue and will provide an update when there is an ETA for correction of the issue.

Transactions using a token affected by this issue will receive the message "COULD NOT LOCATE ORIGINAL TRANSACTION". They will need to be reattempted after the issue is resolved.

INVESTIGATING 5 months ago - at 06/20/2024 05:22AM

Additional data has been found to be missing. We will be working to correct the database synchronization issue and restore all the missing data.

MONITORING 5 months ago - at 06/19/2024 04:07PM

Connectivity issue has been corrected and data merged. Data from all transactions is now available.

IDENTIFIED 5 months ago - at 06/19/2024 03:52PM

A connectivity issue between BluePay database servers caused transaction data from transactions processed 6/18/2024 4:16pm-4:26pm EDT to not be merged to all database servers. Affected transactions are currently not available in the BluePay website and not available from reporting APIs. We are currently working to correct the issue. Once connectivity is restored between the database servers, the transaction data will be merged making it available in the BluePay website and reporting APIs. The transactions will then settle in the next regularly scheduled settlement.

The best way to monitor CardConnect and all your third-party services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3273 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime