2026/06/10 - Native Fax Inbound and Outbound Failures (Resolved)
Table of Contents
Customer Impact (Summary)
We understand how disruptive a fax outage is to the businesses that rely on it, and we apologize for the impact this caused.
- What happened: Customers were unable to send or receive faxes.
- Who was affected: All Native Fax clients.
- When it began: June 10, 2026 at 5:37 PM ET.
- Total downtime: Approximately 13.5 hours across two separate periods (see Impact Summary).
- Current status: Resolved. Fax service has been stable since the evening of June 11 and was confirmed fully resolved on June 16, 2026.
- Workaround: None was available during either outage, as communicated in the status announcements.
Affected Services
- Native Fax: both inbound and outbound
Event Summary
On June 10, 2026 at 5:37 PM ET, the Native Fax service that processes inbound and outbound faxes crashed and did not recover on its own, causing all faxes to fail. After working with the fax platform vendor through the evening — including an emergency upgrade of the fax software — service was restored and confirmed working at 11:35 PM ET. At that point we believed the issue was fully resolved.
The following day, June 11 at 12:41 PM ET, inbound faxes began failing again. This was a separate, previously unknown software bug. Our engineering team and the vendor's senior engineers worked through the afternoon to identify and fix the underlying cause. Service was restored at 8:00 PM ET and confirmed stable, with monitoring beginning at 8:14 PM ET. Service remained stable through the weekend and the following week.
Event Timeline
June 10, 2026
Alerting / Identification
- 5:37 PM ET — First reports of faxes failing to send or receive; our engineering team engaged.
- 5:51 PM ET — Our engineering team restarted the fax service, which showed online again.
- 5:59 PM ET — Test faxes were still failing, so the vendor was engaged for assistance.
Investigation
- 6:30 PM ET — Our engineering team and the vendor joined a call. Database connection errors were found and corrected. Faxes could then send but not receive, which was traced to the fax ports being stuck.
Working Resolution
- 7:53 PM ET — The vendor began installing a new version of the fax software.
- 8:15–10:09 PM ET — The upgrade repeatedly failed due to low disk space and a full database log file on the fax server. Our engineering team cleared older temporary files and the database log to free up space, which allowed the upgrade to continue.
- 10:55 PM ET — The upgrade completed and services restarted. Inbound and outbound test faxes were successful.
- 11:15 PM ET — Additional test faxes failed with a "fax service busy" error.
- 11:27 PM ET — The vendor adjusted the configuration and restarted the service.
Monitoring
- 11:35 PM ET — Inbound and outbound test faxes were successful and monitoring began.
June 11, 2026
Monitoring (continued) / Recurrence
- 11:02 AM ET — No further failures; the fax service was confirmed working correctly.
- 12:41 PM ET — Reports came in of inbound faxes failing with a busy tone. Outbound faxes continued to work. This was caused by a separate, previously unknown software bug.
- 12:51 PM ET — Our engineering team found that internal IP addresses were being incorrectly added to a block list, causing the inbound failures. The vendor was engaged immediately.
Investigation / Working Resolution
- 1:01–1:25 PM ET — The vendor updated the configuration to stop internal IP addresses from being blocked. The fix applied, but the fax service restarted in a stuck state.
- 1:54–2:46 PM ET — Investigation found the stuck state was caused by a different, previously unknown error. Updated software drivers were tried and then rolled back when they prevented the service from restarting. The vendor escalated to its senior engineers. Testing isolated the problem to the fax-receiving side of the service.
- 3:55–4:20 PM ET — A standalone version of the fax software started successfully, but reinstalling the updated software drivers caused another crash, which was again rolled back.
- 4:40–5:10 PM ET — The vendor provided a newer version that already included the fix for the IP address blocking. It installed successfully but still restarted in a stuck state.
- 5:32–6:31 PM ET — The vendor's senior engineers added a debugging tool. The service started and cleared the backlog of queued faxes, and the server was rebooted to clear its memory. After the reboot it returned to the stuck state, which the debugging tool was able to capture.
- 8:00 PM ET — The vendor's senior engineers identified the cause of the stuck state and applied a fix. The service restarted and inbound and outbound faxes completed successfully.
Monitoring
- 8:14 PM ET — Faxes continued to send and receive successfully and monitoring began.
June 12–16, 2026
- Faxes continued to send and receive successfully throughout the weekend and the following week. The incident was considered fully resolved on June 16, 2026 at 9:00 AM ET.
Root Cause
This event involved two related disruptions:
- June 10 (initial outage): A combination of database connection errors and resource limits on the fax server — low disk space and a full database log file — prevented the fax service from running normally. Our team and the vendor performed an emergency upgrade of the fax software, and after configuration adjustments, service was restored. We believed at this point that the issue was fully resolved.
- June 11 (recurrence): A configuration issue caused inbound faxes to be incorrectly blocked. While applying a fix for that issue, a separate and previously unknown software bug was uncovered that caused the fax service to freeze when restarting. This required escalation to the vendor's senior engineering team. Once the underlying bug was identified, it was fixed and the service has remained stable since.
Impact Summary
- Total customer impact: Approximately 13.5 hours across two periods.
- First period: Inbound and outbound faxes were unavailable from June 10 at 5:37 PM ET, with full service restored by 11:35 PM ET (approximately 6 hours).
- Second period: Inbound fax reception failed beginning June 11 at 12:41 PM ET. Full service was confirmed stable at 8:14 PM ET (approximately 7.5 hours).
- All Native Fax clients were affected during both periods. No workaround was available during either outage.
Future Preventative Action
Immediate actions taken:
- During the incident, the vendor's senior engineering team developed new code specifically to resolve the underlying bug, rather than relying on an existing release. This fix — which addresses the original freeze and prevents internal IP addresses from being added to the block list — was delivered and installed during troubleshooting.
Long-term actions:
- Apply the additional patches the vendor has since provided, during an upcoming maintenance window.
- Investigate server improvements where needed.
- Continue working with the vendor to improve Native Fax's resiliency so the service can recover on its own and fail over gracefully without manual intervention.
If you need additional support, please contact support@oit.co.