System-wide Performance Degradation

• 11:00 AM PDT (UTC-7) June 20th 2022

  What happened?
  On June 16th, 2022 a load balancer misconfiguration caused some requests to be routed to non-existent back end services. This caused intermittent failed requests across all parts of the platform.

  What was the impact?
  From ~1:15 pm to ~2:11 pm PDT, ART19 services including episode playback and CMS functionality were intermittently unavailable.

  How did we resolve the issue?
  We reset our load balancer nodes, and updated our configuration. All alerts were cleared and we were fully recovered by 2:19 pm PDT.

  What steps are we taking to prevent future incidents?
  The new configuration we deployed to mitigate the situation on June 16th, 2022, prevents a misconfiguration from having customer facing impact. Additionally, we are researching options to better test and alarm on misconfigurations in our load balancer builds before they happen.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  02:19 PM PDT (UTC-7) June 16th 2022

  We are fully recovered from this issue.

  02:11 PM PDT (UTC-7) June 16th 2022

  We are seeing recovery from the earlier issue. We will continue to closely monitor the situation and are working on configuration updates to prevent this from happening again in the future. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  01:15 PM PDT (UTC-7) June 16th 2022

  We’re actively investigating an intermittent issue affecting listener access to content, as well as the ART19 CMS. As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.