Stats Delay
-
9:16 AM PDT (UTC-7) September 11th, 2020
What happened?
On 9/8/20 at 4:24 am PDT (UTC-7), stats processing began to be delayed.
What was the impact?
All stats, including episode downloads and campaign impressions, were not up to date for end-users of ART19 for approximately 12 hours (4 am - 4 pm PDT).
In addition, 3rd party tracking event processing was paused for approximately 3 hours (12:45 pm - 3:50 pm PDT).
No data was lost during this time.
How did we resolve the issue?
At 10:30 am PDT the root cause was identified, and work began to resolve the issue.
By approximately 4 pm PDT stats were recovered.
By 8:18 pm PDT all 3rd party tracking events were fully executed.
What steps are we taking to prevent future incidents?
We separated processing by event type to ensure that performance issues with one type would not affect another.
We also improved our instrumentation to diagnose future performance issues more quickly.
Finally, we are working with our provider to make performance and reliability improvements and monitoring 3rd party tracking providers to determine if other actions need to be taken.
Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.
6:45 PM PDT (UTC-7) September 8th, 2020
The issue affecting stats has been resolved. In order to resolve this issue, external tracking events were not processed for several hours. There was no data loss, but it may take up to an hour for the backlogged events to finish processing. We will provide a postmortem later this week. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.
2:40 PM PDT (UTC-7) September 8th, 2020
We are actively working on a resolution for a stats delay affecting all customers. There is no data loss, and we apologize for any inconvenience.
If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.