System-wide Performance Degradation

• 9:47 AM PDT (UTC-7) May 9th 2022

  What happened?
  On May 5th, 2022, a significant increase in listener traffic, combined with a number of active ad campaigns with complex targeting configurations, caused the ad decisioning engine to put a heavy load on our application database.

  What was the impact?
  From ~6:49 am to ~9:55 am PDT, ART19 services including episode playback and CMS functionality were intermittently unavailable. Third-party ad tracking pings were also delayed by approximately 1 hour until later in the afternoon.

  How did we resolve the issue?
  We paused a number of campaigns at approximately 8:30 am PDT to free up database resources. This brought ART19 services back online for listeners and customers. We then worked with affected ad ops teams to optimize campaign settings before reactivation. All affected campaigns were resumed by 12:25 pm PDT.

  What steps are we taking to prevent future incidents?
  We are improving our ad decisioning technology to accommodate trends in system traffic and campaign configuration.
  In the interim, we have updated the guidlines for our support team to respond more quickly to similar incidents.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  1:05 PM PDT (UTC-7) May 5th 2022

  The system degradation has been resolved. We will provide a postmortem within the next few days. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  9:55 AM PDT (UTC-7) May 5th 2022

  The root cause of the system degradation has been identified and as of 9:55 am PDT all services are fully available. We continue to closely monitor the situation as traffic returns to normal. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  6:53 AM PDT (UTC-7) May 5th 2022

  ART19 is currently experiencing system wide performance degradation. We’re actively investigating and working on a resolution. As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.