WarpFeed Transcoder Degradation

• 12:05 AM PDT (UTC-7) Dec 07 2022

  What happened?
  On 12/5/22 at 02:05 PM PDT we experienced an issue to the Public Transcoder responsible for generating episode combination downloads. Our load balancer encountered an increased rate of memory usage causing all available memory to be used up.

  What was the impact?
  Listeners may have experienced latency issues in episode downloads between approximately 02:05 PM PDT and 4:25 PM PDT.

  How did we resolve the issue?
  We improved the configuration of and replaced the hardware used by the Load Balancers to have more memory available, providing more stability and better container utilization.

  What steps are we taking to prevent future incidents?
  We have a planned refactor of our Load Balancer that handles traffic to the Public Transcoder to use a less resource-intensive solution.

  We apologize for any inconvenience caused. Feel free to reach out to support@art19.com with any other questions or concerns.

  05:15 PM PST (UTC-8) Dec 05 2022

  We are recovered from this issue and will publish a postmortem within the next few days.

  05:03 PM PST (UTC-8) Dec 05 2022

  We are seeing recovery from the earlier issue. We will continue to closely monitor the situation and are working on configuration updates to prevent this from happening again in the future. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.

  02:09 PM PST (UTC-8) Dec 05 2022

  We’re actively investigating an issue affecting the WarpFeed transcoder. As soon as we know more we will send a follow up. We’re sincerely sorry for the inconvenience and appreciate your patience. If you have any questions or concerns don’t hesitate to reach out to us at support@art19.com.