Live Status\Incident History

January 11, 2024

Healthcheck Assertion Failed

Minor Outage

Opened: Jan 11, 2024, 10:52 AM UTC

Duration: 166h 17m 36s


  • Opened
    Jan 11, 2024, 10:52 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jan 11, 2024, 10:56 AM UTC

    Issue resolved on monitor recovery.

  • Unresolved
    Jan 11, 2024, 11:00 AM UTC

    Issue re-opened on monitor failure.

  • Identified
    Jan 11, 2024, 12:09 PM UTC

    Multiple undersea cable cuts have resulted in various important paths between Singapore and other PoPs becoming degraded. As such, our Singapore ingests have been rerouted to other regions until the situation improves.

  • Resolved
    Jan 18, 2024, 9:13 AM UTC

    Issue resolved on monitor recovery.

January 2, 2024

Healthcheck Assertion Failed

Minor Outage

Opened: Jan 2, 2024, 12:09 PM UTC

Duration: 72h 59m 58s


  • Opened
    Jan 2, 2024, 12:09 PM UTC

    Issue opened on monitor failure.

  • Identified
    Jan 3, 2024, 1:11 AM UTC

    Our Miami PoP is offline due to network isolation (private waves are offline). Traffic is being rerouted to other regions for the time being.

  • Resolved
    Jan 5, 2024, 1:09 PM UTC

    Issue resolved on monitor recovery.

October 4, 2023

New York Ingest Offline

Minor Outage

Opened: Oct 4, 2023, 8:30 AM UTC

Duration: 12h 20m 45s

Affected services:Ingest New York - stream.ewr.irl.run


  • Identified
    Oct 4, 2023, 8:30 AM UTC

    Ingest server hardware keeps locking up, waiting on replacement. If you are experiencing issues switch to stream.global.irl.run

  • Resolved
    Oct 4, 2023, 8:50 PM UTC

    Replacement hardware has been installed and the ingest has been put back into the pool.

September 21, 2023

Healthcheck Assertion Failed

Minor Outage

Opened: Sep 21, 2023, 5:50 PM UTC

Duration: 146h 29m 55s


  • Opened
    Sep 21, 2023, 5:50 PM UTC

    Issue opened on monitor failure.

  • Identified
    Sep 21, 2023, 5:53 PM UTC

    We've identified some routing-related connectivity issues with our Amsterdam ingest location. Streams being served by this ingest have been temporarily rerouted to the next closest locations.

  • Resolved
    Sep 27, 2023, 8:20 PM UTC

    Issue resolved on monitor recovery.

July 25, 2023

Healthcheck Assertion Failed

Minor Outage

Opened: Jul 25, 2023, 7:33 AM UTC

Duration: 15h 5m 19s


  • Opened
    Jul 25, 2023, 7:33 AM UTC

    Issue opened on monitor failure.

  • Identified
    Jul 25, 2023, 7:45 AM UTC

    One of our upstream providers has experienced a hardware fault in a switch serving our hardware in NYC. The region is currently rerouted and minimal impact is expected to be observed.

  • Resolved
    Jul 25, 2023, 10:39 PM UTC

    Issue resolved on monitor recovery.

June 22, 2023

LAX Colo Network Outage

Outage

Opened: Jun 22, 2023, 9:20 PM UTC

Duration: 0s

Affected services:Dashboard - irl.run


  • Identified
    Jun 22, 2023, 7:00 AM UTC

    Incident began

  • Resolved
    Jun 22, 2023, 7:30 AM UTC

    Between approximately 12:00AM PST and 12:30AM PST, an outage occurred which affected nearly all of our online users. Long time service members will know that incidents of this scale are incredibly rare, and have only happened a few times in the history of our service. The effects of this outage were wide-ranging, and include any of the following conditions: - Dashboard control unusable - Streams to active destinations failing (causing VOD splits) - Ingest feeds going black and not coming back This incident happened due to a bug in the firmware of our two core routers which was exposed when we attempted a seemingly harmless configuration change. This led to a loss of communication between the two routers. Due to maintenance upstream of us, one of our two main network uplinks were already shut down. When the loss of communication happened between our routers, it resulted in traffic which was previously being rerouted due to the maintenance to instead be blackholed (sent to nowhere). This is what resulted in the affects described above. We immediately worked to begin troubleshooting the issues, and with the help of our upstream provider, were able to get the second upstream link online temporarily in order to minimize the loss of communication between the two core routers. This allowed us enough connectivity to migrate all online users off of the LAX PoP and onto a different one. By 12:30AM, all users were migrated to other PoPs and were able to continue their broadcasts. We have taken immediate action to prevent an issue of this kind from happening again. We completed a pending infrastructure migration which would have allowed us to resolve the issue within only a few minutes, and we have begun seeking alternative hardware vendors to upgrade the two core routers. For any further questions or concerns, please reach out to me at tt2468@irltoolkit.com - Kyle (tt2468) | CEO/Co-founder

June 7, 2023

Dashboard Access Issues

Minor Outage

Opened: Jun 7, 2023, 9:46 AM UTC

Duration: 0s

Affected services:Dashboard - irl.run


  • Monitoring
    Jun 6, 2023, 4:16 AM UTC

    We experienced an issue with part of our network which affected customers' ability to access our dashboard. This period lasted roughly 3.5 hours while we worked to troubleshoot a failing software component and replace it.

  • Resolved
    Jun 7, 2023, 7:30 AM UTC

    The issue is resolved as of 7:30AM UTC

May 23, 2023

Healthcheck Assertion Failed

Minor Outage

Opened: May 23, 2023, 12:26 AM UTC

Duration: 3m 55s


  • Opened
    May 23, 2023, 12:26 AM UTC

    Issue opened on monitor failure.

  • Resolved
    May 23, 2023, 12:30 AM UTC

    Issue resolved on monitor recovery.

  • Resolved
    May 23, 2023, 12:34 AM UTC

    A route leak within one of our upstream carriers caused a 3 minute outage of our New York City ingest. Streams were rerouted automatically as intended.

May 2, 2023

Ingest Maintenance

Maintenance

Posted: May 2, 2023, 3:58 AM UTC

Affected services:Ingest Los Angeles - stream.lax.irl.runIngest New York - stream.ewr.irl.runIngest Singapore - stream.sin.irl.runIngest Tokyo - stream.tyo.irl.runIngest Sydney - stream.syd.irl.run


  • Monitoring
    May 2, 2023, 3:57 AM UTC

    We are performing maintenance to most ingests. Observed effects should be limited to slightly higher SRT RTT while the maintenance is performed.

  • Resolved
    May 2, 2023, 8:49 PM UTC

April 4, 2023

Global Ingest Maintenance

Maintenance

Posted: Apr 4, 2023, 8:55 PM UTC


  • Monitoring
    Apr 4, 2023, 8:53 PM UTC

    Some ingests are rerouted while we perform software maintenance. No significant impact to services is expected.

  • Resolved
    Apr 4, 2023, 11:05 PM UTC

Rotterdam Ingest Issues

Minor Outage

Opened: Apr 4, 2023, 4:10 PM UTC

Duration: 4h 24m 18s


  • Opened
    Apr 4, 2023, 4:10 PM UTC

    Issue opened on monitor failure.

  • Investigating
    Apr 4, 2023, 4:27 PM UTC
  • Identified
    Apr 4, 2023, 6:55 PM UTC

    Our upstream provider responsible for our Rotterdam ingest is currently experiencing an outage. Traffic has been redirected where necessary. No significant service impact is expected.

  • Resolved
    Apr 4, 2023, 8:35 PM UTC

    Issue resolved on monitor recovery.

March 25, 2023

Ingest Maintenance

Maintenance

Posted: Mar 25, 2023, 12:22 AM UTC


  • Opened
    Mar 25, 2023, 12:21 AM UTC

    We are performing maintenance on all ingest servers. Connections have been rerouted where necessary, and no impact to service is expected.

  • Resolved
    Mar 25, 2023, 2:14 AM UTC