PandaDoc outage on Apr 7, 2023

What happened

PandaDoc reported a major outage starting Apr 7, 2023, 21:01 UTC, affecting Creating and editing documents, Sending documents, Uploading documents, Public (recipient) view, Signup, CRMs & Integrations, API, Webhooks and 3 more. It was marked resolved at Apr 8, 2023, 09:23 UTC.

• The incident lasted 12h 22m from first report to resolution.
• It was the 2nd longest of 14 significant outages PandaDoc reported in the 12 months up to it.
• PandaDoc had reported 4 other partial or major outages in the 90 days before it.

Latest update from PandaDoc

## A summary of what happened At **14:01 PDT Friday, April 7th** our monitoring indicated that our Public API request rate dropped and health checks didn’t pass through. The situation deteriorated rapidly and we noticed that some of our API endpoints became unresponsive, which impacted the availability of the PandaDoc platform.  We followed our protocol and immediately started our incident response procedure, rolled back recent updates, and involved engineers in multiple investigation paths. After we had dismissed some initial theories, we understood that we had an issue connected with something on the infrastructure level and started investigating this together with our cloud provider \(AWS\). After a deep investigation that lasted several hours, we were able to track down the issue to network problems: several pods on a specific Kubernetes node were experiencing intermittent low-level network issues that caused connection leaks - repeatedly opening connections without closing them

Read PandaDoc's official incident report ↗

What our independent monitoring saw

Our independent monitoring of PandaDoc started after this incident, so this report is based on PandaDoc's status page.

PandaDoc's overall reliability

Full PandaDoc reliability report and live status → · Most reliable PandaDoc alternatives →