Media Upload Issue

• 3:30 PM PDT (UTC-7) April 29 2020

  What happened?
  On 4/28/2020 at approximately 6:15 PM PDT (UTC-7) a large operation was initiated to process thousands of episodes, taxing our transcoder resources. The alarms set up to detect this situation were not correctly configured to trigger an on-call response.

  What was the impact?
  Some users were unable to upload new ads and content to the ART19 platform for approximately 6 hours.

  End users were not impacted and were still able to access published episodes and feeds.

  How did we resolve the issue?
  We scaled up infrastructure to handle the processing load and implemented measures to limit the number of episodes being processed at a time.

  What steps are we taking to prevent future incidents?
  We are putting multiple protections in place, including updating infrastructure to maximize available disk space, adding controls to throttle large transcoding jobs in progress, and improving our monitoring and alarm configuration to expedite incident resolution.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  12:35 AM PDT (UTC-7) April 29 2020

  The issue affecting uploads of new ads and content to the ART19 platform has been resolved. We will provide a postmortem later this week. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  9:30 PM PDT (UTC-7) April 28 2020

  We are actively investigating an issue impacting uploads of new ads and content to the ART19 platform.
  This does not affect end users, who can still access all published episodes and feeds. We apologize for any inconvenience.

  If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.