https://irltk.cronitorstatus.com/historyStatus Page - Incident Feeden-usThu, 11 Jan 2024 10:52:17 +0000https://irltk.cronitorstatus.com/history/33e8bf8f36836dd0[Unresolved] 2024-01-11 10:52 UTC - "Issue opened on monitor failure." [Resolved] 2024-01-11 10:56 UTC - "Issue resolved on monitor recovery." [Unresolved] 2024-01-11 11:00 UTC - "Issue re-opened on monitor failure." [Identified] 2024-01-11 12:09 UTC - "Multiple undersea cable cuts have resulted in various important paths between Singapore and other PoPs becoming degraded. As such, our Singapore ingests have been rerouted to other regions until the situation improves." [Resolved] 2024-01-18 09:13 UTC - "Issue resolved on monitor recovery."Thu, 11 Jan 2024 10:52:17 +0000https://irltk.cronitorstatus.com/history/33e8bf8f36836dd0https://irltk.cronitorstatus.com/history/32fb7edda8718c4d[Unresolved] 2024-01-02 12:09 UTC - "Issue opened on monitor failure." [Identified] 2024-01-03 01:11 UTC - "Our Miami PoP is offline due to network isolation (private waves are offline). Traffic is being rerouted to other regions for the time being." [Resolved] 2024-01-05 13:09 UTC - "Issue resolved on monitor recovery."Tue, 02 Jan 2024 12:09:28 +0000https://irltk.cronitorstatus.com/history/32fb7edda8718c4dhttps://irltk.cronitorstatus.com/history/794f644d1aaad59c[Identified] 2023-10-04 08:30 UTC - "Ingest server hardware keeps locking up, waiting on replacement. If you are experiencing issues switch to stream.global.irl.run" [Investigating] 2023-10-04 12:29 UTC [Identified] 2023-10-04 12:29 UTC [Resolved] 2023-10-04 20:50 UTC - "Replacement hardware has been installed and the ingest has been put back into the pool."Wed, 04 Oct 2023 08:30:02 +0000https://irltk.cronitorstatus.com/history/794f644d1aaad59chttps://irltk.cronitorstatus.com/history/746bd44c24758327[Unresolved] 2023-09-21 17:50 UTC - "Issue opened on monitor failure." [Identified] 2023-09-21 17:53 UTC - "We've identified some routing-related connectivity issues with our Amsterdam ingest location. Streams being served by this ingest have been temporarily rerouted to the next closest locations." [Resolved] 2023-09-27 20:20 UTC - "Issue resolved on monitor recovery."Thu, 21 Sep 2023 17:50:32 +0000https://irltk.cronitorstatus.com/history/746bd44c24758327https://irltk.cronitorstatus.com/history/c2f61e0b970106dc[Unresolved] 2023-07-25 07:33 UTC - "Issue opened on monitor failure." [Identified] 2023-07-25 07:45 UTC - "One of our upstream providers has experienced a hardware fault in a switch serving our hardware in NYC. The region is currently rerouted and minimal impact is expected to be observed." [Resolved] 2023-07-25 22:39 UTC - "Issue resolved on monitor recovery."Tue, 25 Jul 2023 07:33:41 +0000https://irltk.cronitorstatus.com/history/c2f61e0b970106dchttps://irltk.cronitorstatus.com/history/e31f49739336d1c8[Identified] 2023-06-22 07:00 UTC - "Incident began" [Resolved] 2023-06-22 07:30 UTC - "Between approximately 12:00AM PST and 12:30AM PST, an outage occurred which affected nearly all of our online users. Long time service members will know that incidents of this scale are incredibly rare, and have only happened a few times in the history of our service. The effects of this outage were wide-ranging, and include any of the following conditions: - Dashboard control unusable - Streams to active destinations failing (causing VOD splits) - Ingest feeds going black and not coming back This incident happened due to a bug in the firmware of our two core routers which was exposed when we attempted a seemingly harmless configuration change. This led to a loss of communication between the two routers. Due to maintenance upstream of us, one of our two main network uplinks were already shut down. When the loss of communication happened between our routers, it resulted in traffic which was previously being rerouted due to the maintenance to instead be blackholed (sent to nowhere). This is what resulted in the affects described above. We immediately worked to begin troubleshooting the issues, and with the help of our upstream provider, were able to get the second upstream link online temporarily in order to minimize the loss of communication between the two core routers. This allowed us enough connectivity to migrate all online users off of the LAX PoP and onto a different one. By 12:30AM, all users were migrated to other PoPs and were able to continue their broadcasts. We have taken immediate action to prevent an issue of this kind from happening again. We completed a pending infrastructure migration which would have allowed us to resolve the issue within only a few minutes, and we have begun seeking alternative hardware vendors to upgrade the two core routers. For any further questions or concerns, please reach out to me at tt2468@irltoolkit.com - Kyle (tt2468) | CEO/Co-founder"Thu, 22 Jun 2023 21:20:45 +0000https://irltk.cronitorstatus.com/history/e31f49739336d1c8https://irltk.cronitorstatus.com/history/c7b7034538b4775e[Monitoring] 2023-06-06 04:16 UTC - "We experienced an issue with part of our network which affected customers' ability to access our dashboard. This period lasted roughly 3.5 hours while we worked to troubleshoot a failing software component and replace it." [Resolved] 2023-06-07 07:30 UTC - "The issue is resolved as of 7:30AM UTC"Wed, 07 Jun 2023 09:46:47 +0000https://irltk.cronitorstatus.com/history/c7b7034538b4775ehttps://irltk.cronitorstatus.com/history/3d618f7844719802[Unresolved] 2023-05-23 00:26 UTC - "Issue opened on monitor failure." [Resolved] 2023-05-23 00:30 UTC - "Issue resolved on monitor recovery." [Resolved] 2023-05-23 00:34 UTC - "A route leak within one of our upstream carriers caused a 3 minute outage of our New York City ingest. Streams were rerouted automatically as intended."Tue, 23 May 2023 00:26:55 +0000https://irltk.cronitorstatus.com/history/3d618f7844719802https://irltk.cronitorstatus.com/history/5847d8183ad2274a[Monitoring] 2023-05-02 03:57 UTC - "We are performing maintenance to most ingests. Observed effects should be limited to slightly higher SRT RTT while the maintenance is performed." [Resolved] 2023-05-02 20:49 UTCTue, 02 May 2023 03:58:35 +0000https://irltk.cronitorstatus.com/history/5847d8183ad2274ahttps://irltk.cronitorstatus.com/history/b2ff379a1200133e[Monitoring] 2023-04-04 20:53 UTC - "Some ingests are rerouted while we perform software maintenance. No significant impact to services is expected." [Resolved] 2023-04-04 23:05 UTCTue, 04 Apr 2023 20:55:05 +0000https://irltk.cronitorstatus.com/history/b2ff379a1200133ehttps://irltk.cronitorstatus.com/history/76946cf7de97c7d2[Unresolved] 2023-04-04 16:10 UTC - "Issue opened on monitor failure." [Investigating] 2023-04-04 16:27 UTC [Identified] 2023-04-04 18:55 UTC - "Our upstream provider responsible for our Rotterdam ingest is currently experiencing an outage. Traffic has been redirected where necessary. No significant service impact is expected." [Resolved] 2023-04-04 20:35 UTC - "Issue resolved on monitor recovery."Tue, 04 Apr 2023 16:10:54 +0000https://irltk.cronitorstatus.com/history/76946cf7de97c7d2https://irltk.cronitorstatus.com/history/b8fa1e807bef41b2[Unresolved] 2023-03-25 00:21 UTC - "We are performing maintenance on all ingest servers. Connections have been rerouted where necessary, and no impact to service is expected." [Resolved] 2023-03-25 02:14 UTCSat, 25 Mar 2023 00:22:19 +0000https://irltk.cronitorstatus.com/history/b8fa1e807bef41b2