Terra - Degraded API performance and delayed data delivery – Incident details

All systems operational

Degraded API performance and delayed data delivery

Resolved
Degraded performance
Started about 2 months agoLasted about 3 hours

Affected

Terra Infrastructure

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

REST endpoints

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

Event Delivery

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

Updates
  • Resolved
    UTC
    Resolved
    Service has recovered. Data during this incident was delayed, not lost, and has been delivered. We are adding stronger saturation alerting and capacity guardrails so this class of issue is caught earlier.
  • Identified
    UTC
    Identified
    We identified saturation in a database used by our connection and configuration service. We are restarting it and increasing its capacity.
  • Investigating
    UTC
    Investigating
    We are investigating degraded API performance. Some requests are returning errors or timing out, and some data deliveries are delayed.