Issues Making/Receiving calls

Dear Valued Customer,

We are writing to sincerely apologize for the service outage you experienced today. We want to provide full transparency on what happened and the steps we are taking to prevent recurrence.

Over the last few months, we have completely overhauled our infrastructure and equipment to support advanced new technologies, including AI analysis and more sophisticated automated systems. To put this in perspective, we have implemented more upgrades in the last year than in the previous three years combined.

While we rigorously test all updates in multiple production environments, the scale of these changes has at times caused instability. A full Monday peak workload can expose unforeseen issues that pre-deployment testing cannot always catch—which is exactly what happened today.

Here is what occurred:

• The Incident: Early Sunday morning, we applied a small follow-up update that passed all weekend testing. Systems remained stable until Monday afternoon.
• The Cause: During peak server load, a bug in the new code caused our database to over-protect uncommitted data, blocking new requests from entering the system and effectively halting new calls.
• The Recovery: Our team identified and resolved the issue quickly. However, due to the size of our environment, it took several minutes for all systems to fully recover.

What We Are Doing Next

To ensure that our ongoing upgrades stop causing disruptions, we are changing our deployment protocol immediately:

1. Staged Rollouts: From now on, updates will be deployed in stages across the system rather than applied all at once, allowing us to catch issues before they have a broader impact.
2. Paused Upgrades: We had two additional updates scheduled for the coming month; these have been explicitly postponed until further notice while we stabilize our deployment process.
3. Strict Scheduling: Moving forward, updates will only be applied when absolutely necessary, with added layers of peak-load testing and safeguards.

We are committed to doing better. We are working through the complexity of this modernization so that you benefit from a significantly more capable platform—and we hope the recent upgrades, including our new account portal, AI receptionist, and intelligent call summaries, are already reducing your admin overhead and improving how your callers are handled every day.

Thank you for your patience and continued partnership.

Sincerely,

Telebroad Engineering & Infrastructure Team