Delayed stats processing on some charts

• 12:30 PM PST (UTC-8) January 5th, 2022

  What happened?
  On January 1st, 2022, a database function for returning the current time did not return the year 2022, so new listen data failed to update.

  What was the impact?
  From 12:00 am to ~1:00 pm PT, several charts did not display data for January 1st, and the ART19 ad server made ad decisions based off of stale data. No data was lost.

  How did we resolve the issue?
  We stopped queue processing and restarted the containers that process from that queue. This allowed our database to start reporting on 2022 events again.

  What steps are we taking to prevent future incidents?
  We will continue to investigate how our database server handles processing time during the transition to a new year and address the issue.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  12:09 PM PST (UTC-8) January 1st, 2022

  The stats problem has been resolved for customers. The root cause of the issue is related to a database date function that was returning an incorrect year. Further investigation into a long term fix will continue after the holiday weekend. There was no data loss from this incident.

  10:20 AM PST (UTC-8) January 1st, 2022

  We are actively investigating an issue affecting some of the charts. We’ve discovered a problem in processing that will not cause any data loss. Some charts will have stale data, or no data for January 1st until we resolve the problem. We will post an update as soon as we have the problem resolved.

  We apologize for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.