Affected Services  

• Call Recording

Event Summary  

Call recordings were inaccessible for some time after migrating call recordings and call recording functionality to the new SNAPhosted v2 infrastructure. Additionally, calls that were solely processed by ATL were not being recorded at all after the migration, requiring a service restart during a maintenance window outside of business hours.

Event Timeline

April 14, 2025

• 2:00 AM ET - The migration team finished migrating recording servers from LAS & GRR to IAD & PHX.
• 10:00 AM ET – OIT's Support team was alerted of missing call recordings from the portal.
• 10:30 AM ET – OIT Support confirmed with management and engineering that call recordings were inaccessible from the Manager Portal.
• 10:40 AM ET – OIT Engineering notified the migration team and NetSapiens Support of the missing call recordings.
• 10:52 AM ET – A major incident was declared.
• 12:06 PM ET – NetSapiens Support begins investigating the issue.
• 12:48 PM ET – Recordings for all calls processed by LAS or GRR become accessible in the Manager Portal. OIT Engineering also confirmed that calls that occurred before the migration that were processed by ATL are accessible. However, call recordings for calls processed by ATL from 2 AM ET and on are not.
• 1:33 PM ET – NetSapiens Support is re-engaged for the missing ATL call recordings.
• 1:53 PM ET – A potential fix is put in place on ATL.
• 2:00 PM ET – OIT Engineering confirmed that the attempted fix was unsuccessful.
• 2:42 PM ET – NetSapiens Support and OIT Engineering begin performing a full audit of ATL's call recording setup.
• 4:05 PM ET – NetSapiens Support proposes a service restart as a potential solution.
• 4:49 PM ET – OIT Engineering agrees that a service restart is required.
• 5:19 PM ET – OIT Engineering attempts to put a workaround in place so ATL's calls will be recorded up until the maintenance window.
• 5:25 PM ET – OIT Engineering determines that the attempted workaround was unsuccessful.

April 15th, 2025

• 1:00 AM ET – Maintenance begins.
• 3:06 AM ET – OIT Engineering confirms the service restart has resolved the issue, while also confirming calls that should have been recorded on ATL were not during the incident. However, it is determined that the RTP rebuild from VoIPmonitor is still temporarily available for calls with missing call recordings.
• 9:00 AM ET – OIT Support confirms continued call recording stability, and that the only missing call recordings are for calls processed by ATL on 4/14 2:00 AM - 4/15 3:06 AM.
• 9:15 AM ET - Incident resolution is declared.

Root Cause

While migrating OIT's call recording infrastructure from SNAPhosted v1 to SNAPhosted v2, the migration team didn't update the configuration that decides where the portal downloads call recordings. This resulted in the Manager Portal displaying a "No Recording" message for recorded calls. Additionally, ATL's NetSapiens services were never restarted after the migration, while LAS and GRR's services were, resulting in ATL attempting to send call recordings to the old call recording server until services were restarted.

Impact Summary

Call recordings for calls processed by all servers were unavailable for some time.

Calls processed by ATL after the migration window until the service restart (2:00 AM EST on April 14th, 2025 until 3:06 AM EST on April 15th) were not recorded.

Future Preventative Action

Immediate preventative action taken:

• An update to the configuration files that handle where the Manager Portal pulls call recordings from to ensure changes are reflected on all servers.

Long-term action:

• Improved migration processes and testing to ensure required service restarts and configuration changes are not missed in the future.