Resolved -
A possible firmware/hardware bug was identified, and Teraswitch NOC elected to do a sequential software upgrade on the redundant switch pair in R102.
After the reloads were complete, operations seemed to normalized. We will investigate this network switch pair further to understand the event better.
Please report any outstanding issues with TYO2 to noc@teraswitch.com or to our team directly.
Jul 28, 16:52 UTC
Investigating -
Teraswitch is looking into issues related to the R102 compute rack within TYO2.
Jul 28, 16:17 UTC
Resolved -
Our facility partner has restored power to Teraswitch's SGP2 network racks.
Root cause: both redundant PDUs in each network rack were fed by power breakout boxes connected to the same overhead electrical bus bar, creating a single point of failure. Facility engineers relocated one breakout box to the correct bus, restoring power to the network equipment and eliminating the single point of failure.
We expect no further impact at this time. Customer systems did not lose power at any point during this event.
Jul 18, 01:57 UTC
Identified -
Teraswitch has isolated this incident to a vendor power maintenance event with unexpected impact, affecting both of our network racks and causing an Internet connectivity loss at SGP2. Customer services are believed to be powered on without upstream connectivity. Facility staff are at work to restore Internet connectivity and bring the site back online. We will provide an update as soon as possible.
Jul 18, 00:22 UTC
Investigating -
Teraswitch is investigating reports of Internet connectivity loss at SGP2. We will provide an update shortly.
Jul 17, 23:45 UTC