Duplicate Text Messages Being Received

This incident has been resolved. DLRs are available to all clients who have been requesting them.

We are excited to share an important update regarding Delivery Receipt (DLR) reports. As promised in our last communication, we have continuously worked with our aggregate to resume DLR delivery. We are now pleased to announce that we have made significant progress and are gradually enabling DLR delivery requests for clients who request them.

This process will be incremental as we closely monitor the status to ensure optimal performance and reliability. Our team is dedicated to maintaining the high quality of our messaging services, and we appreciate your patience and understanding throughout this transition.

Stay tuned for more updates as we work towards fully restoring DLR delivery for all clients. Should you have any questions or concerns, please don't hesitate to contact our support team.

Thank you for your continued trust in our services!

Messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We continue to work with our aggregate to resume DLR delivery, and we will provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.

Messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We'll keep monitoring the system to ensure optimal performance and provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.

We're pleased to confirm that the situation remains stable; messaging continues to operate at expected levels. Delivery Receipt (DLR) reports, however, continue to be suspended. We'll keep monitoring the system to ensure optimal performance and provide updates on DLR reports as they become available. Thank you for your patience and understanding throughout this process.

Service levels for messaging continue to operate at expected levels. Please note that Delivery Receipt (DLR) reports remain suspended at this time. We appreciate your ongoing patience and will provide another update in 12 hours.

Our team continues to closely monitor normal service levels for SMS messages following the clearing of the backlog queue. Please note that Delivery Receipt (DLR) reports remain suspended at this time. We appreciate your ongoing patience and will provide another update in 12 hours.

We are pleased to report that the backlog queue has been processed, and there are no longer any delays with inbound messages. Delivery Receipt (DLR) reports remain suspended, and we'll provide further updates as the situation evolves. We appreciate your continued patience and understanding.

We are delighted to announce that the aggregator's patch has been effective, with no errors reported thus far. As of 14:00 UTC (10:00 EST), instances of duplicate messages have significantly decreased, although the backlog queue is still being processed. New outgoing messages might experience temporary delays while we continue clearing the queue. Please be aware that Delivery Receipt (DLR) reports remain suspended for now, and we will share an update once they are back online. Thank you for your patience and understanding during this time.

The aggregate gateway has implemented a solution and we will be closely monitoring its performance with them over the next 72 hours. During this time, their system is processing a backlog of inbound messages, and you may experience some duplicate message deliveries. Please note that new incoming messages may be temporarily delayed as we clear the queue. Delivery Receipt (DLR) reports are currently suspended, and we will provide an update once they resume. We appreciate your patience and understanding.

12:10 PM PST Update: there was a temporary outbound message delay while we troubleshooted the issue with our aggregator. All messages were queued during that period, and the queue has now drained. We are now going to be disabling delivery receipts (DLRs) to continue testing as we are seeing positive momentum towards getting this resolved. Please be advised that you may not receive DLRs for a period of time while we continue working to fix this issue.

10:53 AM PST Update: we are still working with our aggregator to work on correcting this issue, and have been actively on a call with them all morning. We are migrating traffic between binds and performing other tests to find the root cause and get it fixed. Unfortunately, due to the complexity of this issue, implementing a temporary fix at the software layer to prevent duplicate messages is a last-resort option. We will provide an update again later this morning.

1:30 PM PST update: we have been on calls continually with our aggregator today attempting to resolve this ongoing issue. We will continue working through the night and the early AM to get this issue resolved for you. Unfortunately, we don't have an explicit ETA at this time. We can share that issue is transient in nature and is randomly affecting a subset of our overall traffic for all customers. We are performing some deep network analysis (PCAPs, etc.) and queue analysis in tandem with our aggregator to get this fixed as soon as possible. We will provide another update soon.

Firstly, we apologize for the continued inconvenience. We continue working with our aggregator to fix this issue. We have been in constant contact with their technical team, and the root issue stems from our aggregator's side - not TSG Global. We will provide a full postmortem once the issue is resolved. If you have any questions, please reach out to support@tsgglobal.com

A fix has been implemented and we are monitoring throughout the evening for any new issues. We will resolve this incident in the AM on 4/6/2023 if no further issues are encountered. Again, we appreciate your patience as we worked with our aggregator to resolve the issue, and will provide a full postmortem once we collect all the necessary details.

Firstly, we apologize for the inconvenience. We are continuing to work with our aggregator to fix this issue via Zoom. We have been in constant contact with their technical team, and the root issue is stemming from our aggregator's side - not TSG Global. We will provide a full postmortem once the issue is resolved. If you have any questions, please reach out to support@tsgglobal.com

We have reports from customers that duplicate text messages are being received. We are aware of the issue and are working with our aggregator to resolve the issue/implement a fix. It is intermittent, and will have an update soon.