Terra - Degraded API performance and delayed data delivery – Incident details

Degraded API performance and delayed data delivery

ResolvedDegraded performance2 hours 50 minutes

Affected

Terra Infrastructure

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

REST endpoints

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

Event Delivery

Operational from 7:05 PM to 8:15 PM, Degraded performance from 8:15 PM to 9:55 PM

  • ResolvedUTC
    Service has recovered. Data during this incident was delayed, not lost, and has been delivered. We are adding stronger saturation alerting and capacity guardrails so this class of issue is caught earlier.
  • IdentifiedUTC
    We identified saturation in a database used by our connection and configuration service. We are restarting it and increasing its capacity.
  • InvestigatingUTC
    We are investigating degraded API performance. Some requests are returning errors or timing out, and some data deliveries are delayed.