System Outage

• 12:30 PM PDT (UTC-7) September 26th 2019

  What happened?
  On 9/25/19 at 3:20 PDT (UTC-7) we deployed a change that unintentionally invalidated all RSS feed caches and overloaded our application’s database rebuilding the feeds.

  What was the impact?
  All ART19 services, including RSS Feeds, Web Players and our customer CMS, were unavailable for most users between approximately 3:22 and 4:02 PDT.

  How did we resolve the issue?
  We added capacity to our application’s database to recover the application as quickly as possible. We then monitored all systems to ensure they were stable before declaring the outage resolved.

  What steps are we taking to prevent future incidents?
  We are introducing improved error handling so that jobs are better optimized and the impact is mitigated should a similar event occur in the future.

  Again, we apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  4:02 PM PDT (UTC-7) September 25th 2019

  ART19 has recovered from the earlier incident. We’ll be conducting a rigorous investigation and will provide a postmortem. We apologize again for any inconvenience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  3:22 PM PDT (UTC-7) September 25th 2019

  ART19 is currently experiencing a system wide outage impacting our CMS and RSS feeds. We’re actively investigating and working on a resolution. As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.