Resolved -
Our facility partner has restored power to Teraswitch's SGP2 network racks.
Root cause: both redundant PDUs in each network rack were fed by power breakout boxes connected to the same overhead electrical bus bar, creating a single point of failure. Facility engineers relocated one breakout box to the correct bus, restoring power to the network equipment and eliminating the single point of failure.
We expect no further impact at this time. Customer systems did not lose power at any point during this event.
Jul 18, 01:57 UTC
Identified -
Teraswitch has isolated this incident to a vendor power maintenance event with unexpected impact, affecting both of our network racks and causing an Internet connectivity loss at SGP2. Customer services are believed to be powered on without upstream connectivity. Facility staff are at work to restore Internet connectivity and bring the site back online. We will provide an update as soon as possible.
Jul 18, 00:22 UTC
Investigating -
Teraswitch is investigating reports of Internet connectivity loss at SGP2. We will provide an update shortly.
Jul 17, 23:45 UTC
Resolved -
Teraswitch monitoring reports that all impacts are currently resolved. Please report any further issues to our team.
At EWR2, a device sitting between our data center fabric and our internet edge routers experienced a hardware fault that triggered an ASIC-level reload. After the reload, the device came back online in a degraded state and was unable to forward traffic from the data center fabric to the internet edge routers (specifically, it could not process VXLAN-routed traffic).
EWR2 operates six of these core routers. Five were unaffected; however, the impaired unit failed to signal that it was incapable of forwarding traffic. Because downstream routing protocols did not detect this bad state, traffic that landed on the affected device was blackholed.
Our team manually removed the router from the pool of forwarding destinations for this traffic, restoring service. We will perform all necessary follow-up steps to fully resolve the underlying issue and prevent recurrence.
Jul 8, 18:20 UTC
Update -
We have located an unstable core router in EWR2 and have implemented mitigations to divert traffic away from it.
Jul 8, 18:01 UTC
Identified -
We have identified the issue affecting network connectivity in our EWR2 (Newark, NJ) location and are implementing corrective changes.
Jul 8, 18:01 UTC
Investigating -
Teraswitch is investigating reports of intermittent network connectivity issues affecting some services in our EWR2 (Newark, NJ) location.
Jul 8, 17:48 UTC