Stats Are Delayed or Unavailable

• 01:51 PM PST (UTC-8) December 25th, 2022

  What happened?
  On 12/24/22 at 10:32 PM PST our data warehouse provider, used in statistics processing, experienced a hardware failure. After root cause analysis, our vendor replaced the failed hardware.

  What was the impact?
  Between 12/24/22 10:32PM PST and 12/25/22 02:11AM PST, the following impact could have been experienced:

  • Charts had stale data or no data at all,
  • A delay sending events to external tracking servers
  • Ad selection on downloads could have been based on stale statistics

  How did we resolve the issue?
  After talking with our data warehouse provider, we had the degraded node replaced with a new node.

  What steps are we taking to prevent future incidents?
  We have a planned refactor of our stats processing pipeline to provide more real time data and provide more resiliency against our data warehouse being down.

  We apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  03:00 AM PST (UTC-8) December 25th, 2022

  The stats delay is resolved. We are awaiting more information from our data warehouse provider to provide root cause analysis.

  01:00 AM PST (UTC-8) December 25th, 2022

  We have asked our data warehouse provider to identify the root cause of the outage. They were able to make an update to resolve the outage within 30 minutes of our support escalation. Due to the timing of this, we do not expect an explanation to be ready until after the weekend.

  12:00 AM PST (UTC-8) December 25th, 2022

  We are actively investigating a system issue affecting stats processing for all customers. Charts will either have stale data or not load at all until this is resolved. We apologize for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.