pomu.app - Notice history

System Under Maintenance

 
Website
API
CDN/Cache
S3 storage
Production
Dev

Notice history

Apr 2023

No notices reported this month

Mar 2023

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Feb 2023

Livestream recording fails
  • Resolved
    Resolved

    The issue has been resolved. Livestream submission and recording works again.

  • Monitoring
    Monitoring

    We have implemented a fix by updating youtube-dl to the latest version of their master branch, which includes a fix for the issue.

    We are now monitoring if the issue has been resolved or not, which will last until the next livestream in the queue has successfully finished recording and has been archived.

  • Identified
    Identified

    We have identified the issue as being caused by an upstream update to YouTube causing the regex within youtube-dl to no longer successfully parse uploader_id.

    We are working on deploying a fix for this issue.

  • Investigating
    Investigating

    We are currently investigating an incident causing youtube-dl to not be able to list formats for a given stream, resulting in pomu being unable to start the recording process.

    Existing saved livestreams continue to function properly and downloads as well as frontend continues to serve videos. Only new livestreams in the queue are affected.

Dev - Website isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - Website is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - Website cannot be accessed at the moment. This incident was created by an automated monitoring service.

Dev - API isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - API is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - API cannot be accessed at the moment. This incident was created by an automated monitoring service.

Livestream recording fails
  • Resolved
    Resolved

    This incident has been resolved by migrating hosters to fly.io, as all our Hetzner IPs have been banned from YouTube.

    Livestream recording works again. The only issue is that thumbnails do not get saved correctly, however this does not impact the archiving process.

  • Update
    Update

    Our previous fix did not work correctly for livestreams scheduled ahead of time. We have fixed that bug and are now monitoring for further issues.

  • Monitoring
    Monitoring

    We have implemented a fix for the issue by switching from youtube-dl to yt-dlp, as the DASH client implementation is taking longer than expected.

    We are now monitoring for any issues.

  • Identified
    Identified

    We have identified the issue as pomu failing to grab an appropriate playlist url using youtube-dl to pipe to ffmpeg.

    However, pomu is currently being rewritten to support a new file format and at current development progress fixing the existing issue does not make sense as the fix will be replaced soon anyways.

    As a workaround, some important scheduled livestreams such as unarchived karaoke streams will be recorded manually and uploaded to the pomu-manual S3 bucket. The download url for these videos will be fixed up soon after the stream has finished.

  • Investigating
    Investigating

    We are aware of an issue regarding the recording of livestreams and are working on a fix.

Dev - API isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Dev - API is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Dev - API cannot be accessed at the moment. This incident was created by an automated monitoring service.

Feb 2023 to Apr 2023

Next