System Outage

• 12:45 PM PDT (UTC-7) July 16th 2021

  What happened?
  On 7/14/21 at approximately 6 am PDT (UTC-7), customers reported issues accessing the ART19 platform. A misconfiguration of our autoscaling launch setup resulted in insufficient resources to handle the load of increasing Wednesday morning traffic.

  What was the impact?
  For approximately 2 hours, the ART19 platform was slow and in some cases unresponsive for customers. During this time, the success rate of content delivery to listeners dropped by approximately 16%. No data was lost during this time.

  How did we resolve the issue?
  We immediately brought on more resources to remediate the issue. Within 2 hours of reports, customers and listeners were able to consistently access the platform and content. Additionally, we changed the launch configuration of our autoscaling infrastructure to be less prone to timing issues when joining the fleet. This immediately made extra capacity available.

  What steps are we taking to prevent future incidents?
  By 7/16, an improved version of the launch configuration and autoscaling system was deployed. This version eliminates any risk of future timing and configuration issues. We are also adding more monitoring to alert our team when our autoscaling process is not working as intended.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  8:20 AM PDT (UTC-7) July 14th 2021

  The system outage has been resolved. We will provide a postmortem later this week. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  7:00 AM PDT (UTC-7) July 14th 2021

  ART19 is currently experiencing a system wide outage. We’re actively investigating and working on a resolution. As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.