pomu.app - Livestream recording fails – Incident details

Livestream recording fails

Resolved
Partial outage
Started about 2 years agoLasted about 1 month

Affected

Dev

Partial outage from 1:10 AM to 1:27 PM

API

Partial outage from 1:10 AM to 1:27 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved by migrating hosters to fly.io, as all our Hetzner IPs have been banned from YouTube.

    Livestream recording works again. The only issue is that thumbnails do not get saved correctly, however this does not impact the archiving process.

  • Update
    Update

    Our previous fix did not work correctly for livestreams scheduled ahead of time. We have fixed that bug and are now monitoring for further issues.

  • Monitoring
    Monitoring

    We have implemented a fix for the issue by switching from youtube-dl to yt-dlp, as the DASH client implementation is taking longer than expected.

    We are now monitoring for any issues.

  • Identified
    Identified

    We have identified the issue as pomu failing to grab an appropriate playlist url using youtube-dl to pipe to ffmpeg.

    However, pomu is currently being rewritten to support a new file format and at current development progress fixing the existing issue does not make sense as the fix will be replaced soon anyways.

    As a workaround, some important scheduled livestreams such as unarchived karaoke streams will be recorded manually and uploaded to the pomu-manual S3 bucket. The download url for these videos will be fixed up soon after the stream has finished.

  • Investigating
    Investigating

    We are aware of an issue regarding the recording of livestreams and are working on a fix.