(A more detailed/illustrated RCA in PDF format is available on request)

Internet Outage Event at Multiple Sites

Incident date: August 12, 2026  ·  Regions affected: EU, APAC  ·  Full recovery: 33 minutes

Executive summary

In the early morning of August 12, 2026 (UTC), customers at thirteen of Teraswitch's 31 sites lost reachability to Internet destinations and to other Teraswitch sites over the private backbone: twelve across the EU and APAC regions, and MIA1 in North America, which was intentionally and temporarily removed from the backbone as a containment measure. The cause was a stale default route (0.0.0.0/0) originated by an MIA1 edge router: a static route left over from the site's turn-up, redistributed into BGP. The static is deliberately less preferred than any transit-learned default and had sat inactive for months, until a routine transit provider maintenance withdrew the local external default shortly before the first alarm and activated it. As originated, the route carried an empty AS path containing no external networks, a metric of 0, and no communities: the ordinary signature of a redistributed static route.

A digit transposition in a route-map on MIA1's sessions to the global route reflectors, 20326:301:x written where 20326:130:x was intended, then placed the route into the community lists that control regional export, and the global reflectors added the no-export community to the copies sent toward the EU and APAC regions. The route reflector hierarchy propagated it to our EU and APAC markets carrying that flag.

Because the attributes that normally distinguish a remote default from a local one were never attached, receiving edge routers at the affected sites evaluated the route as locally originated and preferred it over their own valid local default. No-export is a control flag that forbids a route from being advertised across a BGP boundary into another ASN. The connection from the edge routers to the data center fabric is exactly such a boundary. When the affected edge routers selected the stale route as their best default, no-export prevented them from advertising any default to their spines. With no default route present, the affected fabrics stopped forwarding Internet-bound traffic, isolating them from their own healthy edge routers.

The first alarm fired at 03:49:52 UTC. Engineers identified the stale route within 10 minutes; the remainder of the outage window was spent executing containment, removing MIA1 from the backbone and powering off the AMS2 route reflector, and waiting for routes to recalculate. Reconvergence began at 04:16:15 UTC, with service returning progressively as sites withdrew the stale default and reconverged on their local default routes; routes fully recalculated and traffic returned to normal 33 minutes after the first alarm. A global configuration change deployed to all compute sites later the same day ensures that a stale or invalid route entering the fabric can no longer prevent traffic forwarding.

Sites affected: 13 of 31 · Time to identify: <10 min · Recovery begins (UTC): 04:16:15 · Alarm to full recovery: 33 min

Customer impact

Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites.

Affected workloads included latency and availability sensitive blockchain infrastructure hosted in these regions, including Solana validator and RPC nodes, which lost Internet and inter-site reachability for the duration of the event. We understand how sensitive and how large these workloads are, and the level of trust their operators place in us.

Impacts under active remediation

Customer DoubleZero's GRE/VPN tunnel sessions were severed when Bare Metal services became unable to reach the DoubleZero devices. We are working to move these interconnects to within the data center fabric, removing their dependency on paths this event was able to break.

Customer load balancers and servers remained able to receive inbound traffic: this was an outbound blackhole, affecting traffic leaving the fabric toward the Internet. Customers who announce their subnets to us via BGP therefore continued to send those announcements, attracting requests into data centers that could not respond to them. This is addressed by the overall hardening, which enforces that the fabric can always reach the Internet whenever its edge routers are otherwise healthy and attracting inbound traffic.

Not affected

North American data centers were not affected.

MIA1 was deliberately taken off the backbone as a containment measure. MIA1 is a small site, and Teraswitch engineering chose to drop it from the backbone quickly to resolve the overall issue. The impact to MIA1 operation was not monitored during this time, but some impact to connectivity at the site would be expected as it moved into island mode, with Internet routing taking local transit only. The site remained in service and was reintegrated into the backbone via a maintenance shortly after the event.

Timeline of events

All times UTC, August 12, 2026.

• ~03:30 A routine transit provider maintenance begins. At an unrecorded moment during this window, likely shortly before the first alarm, er100a.mia1's external default route is withdrawn; the leftover static default activates, enters BGP carrying the transposed community values, receives no-export toward EU and APAC at the global reflectors, and begins propagating.
• 03:49:52 First alarm fires: a monitoring host in LAX1 reports it cannot reach a host in LON1. The stale default route from MIA1 has propagated into EU and APAC, and affected site fabrics stop forwarding Internet traffic.
• +10 min from onset Stale MIA1 default identified as the cause. The responding engineer acted immediately, shutting down the loopback interface carrying MIA1's SR-MPLS/IS-IS backbone connectivity and powering off the AMS2 route reflector, before reporting the cause. The exact moments were not recorded, but fall within this window; the remaining minutes before restoration were consumed by these containment actions and by route recalculation across the affected sites. In the same window the team halted route exports from the global route reflectors. The only operational change from this was peering traffic taking local and regional paths rather than crossing regions; customer and internal routes were maintained by our Continuity Reflectors, which handle only internal and customer routes.
• 04:16:15 Reconvergence begins. With the AMS2 reflector shut down, the backlog of route changes was large, and withdrawing and reprogramming the stale default across the affected fabrics took time. Service returns progressively as each site reconverges on its local default routes.
• ~04:23 Routes fully recalculated and traffic patterns return to normal. Full recovery, 33 minutes after the first alarm.
• 05:18 Global configuration fix decided in coordination with our network hardware vendor; deployment to all compute sites begins.
• by 15:00 Global fabric hardening deployment completed across all compute sites, including unaffected data centers (before 11:00 AM Eastern US). The change was applied one edge router at a time, across the edge routers feeding approximately 62 spines.
• 18:29 Incident marked resolved. Investigation of the underlying condition continues.

Background: how default route signaling works

Each Teraswitch compute site contains a minimum of two Internet/backbone edge routers and two spine routers; larger sites run more of each (four edge routers and six spines, for example). Every edge router connects to every spine, and the spines connect to every leaf switch pair in the data center. Customer bare metal servers attach to the leaf layer.

Under the design in place at the time of the event, the edge routers shared the default route (0.0.0.0/0) they had themselves selected, from local transit or the backbone, into vrf INTERNET on the spines. This default route is the signal by which the leaf/spine network learns that Internet forwarding is available through its local edge routers. It is also the only route the edges send toward the fabric: the spines carry no more specific Internet or inter-site routes, so all traffic leaving a site, whether bound for the Internet or for another Teraswitch data center, follows this single default, and losing it severs both at once. When the local edges originate this export, the next hop is the edge routers themselves, addresses the spines can always resolve directly. Route attributes, specifically communities and metric, distinguish a default originated by the local site edges from one learned from a remote site over the backbone. Each site normally prefers the default originated by its own local edge routers.

Sites learn default routes from other backbone sites by design. If a site loses its own local default, meaning its local edge routers can no longer reach the Internet, the intended behavior is to carry traffic across the backbone to another Teraswitch site as quickly as possible, where healthy edge routers are likely available. The remote defaults exist as that escape path, and the distinguishing attributes exist so the escape path is only chosen when the local path is gone.

Routes move between sites through a redundant route reflection hierarchy: every site normally runs two local route reflectors (MIA1 temporarily did not, as described in the root cause analysis), every region (EU, APAC, US East, US West) has at least two regional reflectors, and three global reflectors tie the regions together. AMS2 hosts one of the EU regional reflectors. Alongside this hierarchy run the Continuity Reflectors: simple route reflectors that hold only Teraswitch data center prefixes and customer prefixes, at a lower priority than the normal reflector fabric, so that a total loss of the normal route reflectors does not cause major route instability or traffic blackholes. As described later, the hierarchy's redundancy could not mitigate this event, because the failure was one of route selection rather than reflector availability: a stale default that appears locally originated wins best path regardless of how many healthy alternatives exist.

Why the default was shared rather than unconditionally originated

This design was deliberate. By sharing a default route the edge had actually selected, rather than always originating one, an edge router could not come online and attract traffic from the spines before it held Internet routes and a usable default of its own. Without this logic, an edge router returning from maintenance or boot could blackhole traffic. The approach served traffic sequencing during maintenance and was believed to provide higher safety and availability.

What happened

A default route originated by MIA1 edge router er100a.mia1 was propagated with an empty AS path containing no external networks (downstream devices in other ASNs would see only AS20326), a metric of 0, no communities of its own, and a next hop pointing at MIA1 router addressing. None of this was stripped in transit: these are the ordinary attributes of a static route redistributed into BGP. The source was a static default route configured during MIA1's turn-up, when the site had a single transit provider; it should have been removed when the second transit was activated, and was not. By design, our edge routers prefer transit-learned defaults over static ones, so the static sat inactive in the background for months, present in configuration but not in use. On August 12, a routine transit provider maintenance beginning around 03:30 UTC withdrew er100a.mia1's external default; the exact moment was not recorded, but the route change likely occurred shortly before the first alarm at 03:49:52 UTC. The static activated and entered BGP. The BGP policy between the two MIA1 edge routers intentionally does not share static routes, so er100b.mia1 never accepted the static default; it continued feeding both MIA1 spines its own working external default, which is why MIA1 itself initially stayed online.

On its way into the route reflector hierarchy, the route was stamped by an inbound policy on MIA1's direct sessions to the global reflectors. That policy contained a digit transposition, 20326:301:x written where 20326:130:x was intended, and the transposed values landed in the community lists that instruct the global reflectors to add the well-known no-export community toward specific regions. The copies exported toward EU and APAC therefore carried no-export; the copy distributed within North America did not, because the corresponding NA value (20326:301:10) was never written by the transposition. This is why the impact was confined to the EU and APAC markets. Within North America the route was present without the flag. Edge routers are largely indifferent to which default wins, and with no flag blocking the edge to spine advertisement of the MIA1 default toward the spine layer, the NA fabrics kept operating as normal. No-export is inert inside our own AS20326, so the route traveled the reflector hierarchy untouched: EU sites received it from the AMS2 regional reflector, APAC sites through the global reflector tier and the APAC regional reflectors. At each affected edge router it won best path, and because the edge to spine connection crosses a BGP boundary into a different ASN, no-export forbade advertising any default across it. The affected fabrics lost their only default.

Failure sequence at each affected site

1. The site's edge routers receive the stale 0.0.0.0/0 from their upstream route reflectors.
2. Because the standard communities that normally identify a remote default were absent, nothing demoted the route on arrival. With local preference equal, best path selection reached the AS path length comparison, where the route's empty AS path (length zero, shown in captures as only the origin code ?) beat the local transit-learned default carrying its provider's ASN. The edge routers selected it over their own valid local default.
3. The route carries the no-export community. The session from the edge routers toward vrf INTERNET on the spine layer crosses a BGP boundary into a different ASN, and no-export forbids the route from being advertised across it.
4. The edge routers therefore stop advertising any default route to the spines. The fabric's only Internet default is withdrawn.
5. With no acceptable default present in vrf INTERNET, the leaf/spine fabric stops forwarding Internet-bound traffic. The fabric is isolated from its own healthy edge routers.

Diagnosis and response

The initial picture was ambiguous. Not all sites were affected, and the pattern appeared geographic, which first pointed the responding engineers toward a DDoS event or a failure of DDoS mitigation systems. Time was also spent investigating whether IS-IS, our internal routing protocol, was involved: roughly a week earlier, a five minute instability had affected the backbone, and a recurrence was a plausible explanation. That earlier event proved to be unrelated. All of this took place within roughly the first ten minutes of the response.

Misleading Spine <-> Edge Traffic

Two observations made the true failure harder to see. First, traffic was still visible flowing from the edge layer to the spine layer at affected sites. The key detail, recognized later, was that this was L2 transport and private VRF traffic. Only the INTERNET vrf had failed, and the presence of healthy traffic on the same links delayed a closer look at the fabric's Internet routing specifically.

Second, management access looked completely normal, for reasons described below.

Management network behavior during the outage

Teraswitch maintains a completely separate management network at each site, fronted by a high availability firewall pair. Under normal conditions these firewalls use the Teraswitch backbone, but they also connect to our serial console servers, which carry a dedicated and isolated Internet connection that is entirely separate from Teraswitch operation. When the data center networks became unusable, the firewalls failed over to this out of band connectivity automatically.

This design worked exactly as intended: our team had unfettered access to every device throughout the outage, as though everything were normal. For the same reason, it was slightly misleading. Because access felt normal, it appeared that nothing major was wrong with the equipment or the fabrics, when in fact the management plane had instantly routed around the very failure under investigation. Good for resolution and access, but misleading for diagnosis.

The moment of identification

Despite the emergency response not recording the broken state of the network, the responding engineer captured the routing state on the AMS3 edge at the moment they located the issue:

er111a.ams3.teraswitch.com#show ip route 0.0.0.0/0

VRF: default
Source Codes:
       C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - Other BGP Routes,
       B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1,
       I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate,
       A O - OSPF Summary, NG - Nexthop Group Static Route,
       V - VXLAN Control Service, M - Martian,
       DH - DHCP client installed default route,
       DP - Dynamic Policy Route, L - VRF Leaked,
       G  - gRIBI, RC - Route Cache Route,
       CL - CBF Leaked Route

Gateway of last resort:
 B I      0.0.0.0/0 [200/0]
           via 100.96.49.1/32, IS-IS SR tunnel index 103
              via TI-LFA tunnel index 40, label 800961
                 via 64.130.60.122, Ethernet25/1, label imp-null(3)
                 backup via 64.130.60.81, Ethernet26/1, label 974060

er111a.ams3.teraswitch.com#show bgp ipv4 unicast
BGP routing table information for VRF default
Router identifier 100.124.63.2, local AS number 20326
Route status codes: s - suppressed contributor, * - valid, > - active, E - ECMP head, e - ECMP
                    S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
                    % - Pending best path selection
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

          Network                Next Hop              Metric  AIGP       LocPref Weight  Path
 * >      0.0.0.0/0              100.96.49.1           0       -          100     0       ? Or-ID: 100.96.63.1 C-LST: 20.32.6.119 20.32.6.1

This is the stale default as received at an affected edge: origin incomplete (the marker of a redistributed route), an AS path containing no external networks (making it appear locally originated), a next hop of 100.96.49.1 in MIA1 addressing, and reflection metadata, originator ID 100.96.63.1 with the AMS2 cluster list, showing the path the route took. The edge itself could resolve that next hop over an IS-IS SR tunnel across the backbone, so from the edge's perspective the route was usable. What made it unusable as the fabric signal was the no-export community it carried: once selected as best, it could not be advertised across the BGP boundary to the spines. Next-hop-self is applied on the edge to spine sessions, so next hop reachability played no part in the fabric failure.

Root cause analysis

Confirmed observations

• The default route was received by EU and APAC edge routers without the standard communities that identify a legitimate remote default, and with an empty AS path containing no external networks: as originated, the bare signature of a locally redistributed static route. As received, it also carried the transposed large-community values described below and the no-export community, established by configuration analysis and lab replication of the full issue.
• The route carried the originator ID of MIA1 edge router er100a.mia1 (100.96.63.1) and a cluster list of 20.32.6.119 20.32.6.1: the AMS2 regional reflector and a global reflector, most recent reflector listed first. This matches MIA1's reflector topology at the time. MIA1's edge routers were connected directly to the global reflectors, a valid but intentionally temporary arrangement used because the edge routers were installed before the site's own reflector units, so no MIA1-local or North American regional entries would be expected on this path.
• At the time of the event, er100b.mia1 was observed properly learning its external default route, and both MIA1 edge routers were receiving defaults from our other regional North American route reflectors. The stale static default sat inactive on er100a.mia1 by design, activating and entering BGP only when er100a.mia1's own external default was withdrawn during the provider maintenance that began around 03:30 UTC.
• Receiving edge routers preferred the route because the attributes that distinguish local origination from remote origination were absent.
• The route carried the no-export community, applied by the global reflectors' regional export policies through the transposed community values. Once an edge router selected this route as its best default, no-export forbade advertising any default across the BGP boundary to the spines, which under the then-current design removed the fabric's only Internet default.
• Next-hop-self is configured on the edge to spine sessions and operated as designed. Next hop reachability played no role in the fabric failure.
• No Teraswitch automation run or configuration change triggered the event; the trigger was an external provider maintenance. The vulnerability it exposed, however, was our own: a standing configuration error, the transposed community values, deployed well before the event.

The default route left on MIA1

The origin of the route was identified in er100a.mia1's configuration: a static default route left over from site turn-up, still present and still redistributed into BGP.

"static_routes": [
    {
        "destination": "0.0.0.0/0",
        "nexthop": "100.105.0.2",
        "vrf": "default"
    }
]

The static was configured when the site operated on a single transit provider and should have been removed when the second transit was activated. Its next hop, 100.105.0.2, is a Dallas edge router directly connected to er100a.mia1. Because our edge routers prefer transit-learned defaults over static ones, it sat inactive in the background for months, doing nothing, until the transit provider maintenance withdrew the external default and activated it.

A same-day audit of every edge router found no other static default routes, and edge router configuration templates have been updated to disallow static default routes entirely. IS-IS has also been blocked from redistributing default and static routes.

How the no-export was applied

Routes from MIA1's edge routers reached the global reflectors over direct sessions, the temporary arrangement described above. An inbound policy on those sessions stamps arriving routes with regional classification values:

route-map RM-NA-INBOUND permit 10
 set large-community 20326:301:30 20326:301:20 20326:130:10 additive

The intended values belong to the 20326:130:x family, which marks normal routes meant to be sent to all regions:

bgp large-community-list standard REGION-NA-EXPORT permit 20326:130:10
bgp large-community-list standard REGION-EU-EXPORT permit 20326:130:20
bgp large-community-list standard REGION-APAC-EXPORT permit 20326:130:30

The first two values written contain a digit transposition: 20326:301:30 and 20326:301:20 where 20326:130:30 and 20326:130:20 were intended. The third, 20326:130:10, was written correctly. The 20326:301:x family is reserved for the opposite purpose, regional export control:

bgp large-community-list standard REGION-NA-NOEXPORT permit 20326:301:10
bgp large-community-list standard REGION-EU-NOEXPORT permit 20326:301:20
bgp large-community-list standard REGION-APAC-NOEXPORT permit 20326:301:30

The global reflectors' export policies toward each region add the well-known no-export community to any route carrying that region's value. The goal of this mechanism is containment, not removal: a route marked this way remains present and usable within AS20326 in those regions, but is never exported to transit or peering there. This is a different case from a separate blocking mechanism in the same policies, which prevents a route from being present in a region at all. The edge to spine sessions cross a BGP boundary of the same kind as transit and peering, which is how a flag aimed at external announcements was also able to suppress the fabric signal. The EU policy is shown; the APAC policy is identical in structure:

route-map RM-EU-OUTBOUND permit 40
  match large-community REGION-EU-NOEXPORT
  set community no-export additive

Every route MIA1 sent over the direct sessions was therefore marked for no-export toward EU and APAC. The NA value, 20326:301:10, was never written by the transposition, so copies distributed within North America carried no flag. This policy is defined for direct edge to reflector sessions and was attached only at MIA1, the only site operating without its own local reflectors. Validation testing at the time did not cover the edge to global reflector operating mode combined with static default routes existing on the edge router; our testing expansion and configuration sanity checking now do.

No capture displaying the route's communities survived from the event window. The chain described here was confirmed by replicating the event in a lab environment using pre-outage configuration state.

The trigger and remaining open items

The origin and the trigger are now confirmed. The leftover static default on er100a.mia1 activated when the provider maintenance described earlier withdrew the local external default. The route entered BGP, was stamped with the transposed community values on MIA1's direct sessions to the global reflectors, and received the no-export community toward EU and APAC as a result. The transposition was not caught in validation testing because testing did not cover the temporary operating mode of an edge router connected directly to the global reflectors; that testing gap has since been closed. Had MIA1 not been removed from the backbone, the event would eventually have resolved itself: when the provider maintenance completed and the external default returned, it would have displaced the static and the route would have been withdrawn.

The interaction between the two MIA1 edge routers is resolved: er100b.mia1 never accepted er100a.mia1's static default route, because the BGP policy between the two edge routers intentionally does not share static routes. This is why MIA1 itself initially stayed online: er100b.mia1 continued feeding both MIA1 spines a working default carrying no no-export flag. One item remains under review: why the EU regional reflector at AMS2 emitted the route toward its clients while the second EU regional reflector, hosted in Frankfurt, did not. The AMS2 reflector's running state, which would have answered this directly, was lost when it was powered off during containment. This document will be revised as it closes.

Contributing factors

• Readiness signaled with external routes rather than a locally originated default. Edge routers did not generate an artificial, clean default route of their own to signal readiness to the data center networks. Instead they re-advertised default routes learned from external transit providers, prepared for that role by route-maps. The readiness signal therefore carried external attributes and inherited external failure modes, and a stale or invalid route could still win best path and stand in for a valid signal.
• Fail-closed fabric behavior. With the edges' best default carrying no-export and therefore barred from the fabric sessions, the fabric was left with no default at all and withdrew Internet forwarding entirely. There was nothing to fall back on: routing does not retain a departed route, and no lower-preference default existed to activate in its place. We considered this a critical design flaw and prioritized its correction. The two available solutions were a lower-preference default that activates when the expected one is lost, or force-advertising the default from the edge routers; we chose force-advertisement, deployed as part of the August 12 hardening.
• Misleading healthy signals during diagnosis. L2 transport and private VRF traffic continued to flow from edge to spine, masking that only the INTERNET vrf had failed, and management access remained fully functional after failing over to out of band connectivity, suggesting equipment and fabrics were healthy. The failover did raise alarms, but within the wave of critical alarms accompanying the onset, nothing specific to the management network rerouting stood out. Together with a geographic pattern resembling a DDoS event and a recent unrelated IS-IS instability, these signals directed early effort away from fabric Internet routing.
• Evidence capture was manual. Service restoration rightly took priority during the response, and debug capture required manual effort. While our responding engineer guessed correctly that the issue was with MIA1's default export, we do not have a capture of the no-export community on the route from the moments during the outage. We supplemented this by replicating the issue in our lab.
• Incorrect generalization from a prior event. A November 2024 default route incident at FRA2 was caused by our own maintenance tooling in combination with a simultaneous route flap at an outside provider. Its remediation led to the assumption that default route failures would originate from within our own change processes, and the design was never evaluated against a stale or invalid update arriving via the backbone itself. See the following section.
• A digit transposition, unprotected by testing. The inbound policy on the direct edge to global reflector sessions was written with 20326:301:x values where 20326:130:x was intended, placing every MIA1 route into the regional no-export control lists. Validation testing did not exercise the temporary operating mode of an edge router connected directly to the global reflectors, so the error was never caught before the event.

Prior related event and design assumptions

In the interest of a complete and honest accounting: this was not the first time a default route problem isolated a Teraswitch data center fabric. Teraswitch was aware of a related failure mode from an incident at FRA2 on November 27, 2024, beginning at 09:35 UTC. The conclusions we drew from that event shaped a design assumption that the August 12, 2026 outage proved incorrect. In both events, a default route carrying no-export won selection on a site's edge routers, and the fabric behind them lost its signal. In both, a routine provider event touching the local default exposed the latent flag.

The November 27, 2024 FRA2 event

The root cause was a combination of unfortunate provider timing and a logical error in the BGP tooling used to divert traffic away from links about to undergo maintenance:

1. The maintenance tooling set a no-export flag (BGP community) on all routes learned from the AS20326 backbone. Inadvertently, this included the default route (0.0.0.0/0) that is exported toward the data center network to signal that an edge router is ready to pass Internet traffic.
2. Initially there was no impact: FRA2's edge routers had selected a default route from a local FRA2 Internet transit provider as that signal.
3. Possibly affected by the same fiber vendor maintenance, one or more local FRA2 transit providers regenerated their default route, causing the FRA2 edge routers to elect a new best default: one generated at a nearby location (FRA1) and learned over the backbone, still carrying the no-export flag.
4. The FRA2 edge routers therefore stopped exporting a default route toward the FRA2 data center network (routers within a different ASN than AS20326). The data center concluded that no edge router was available to process Internet traffic, causing a total FRA2 outage until the flag was removed.

Remediation completed after the 2024 event

• The application of the no-export flag to default routes was removed entirely. It has no traffic steering effect, since Internet destinations almost always have a more specific route within the Teraswitch network.
• A safety was added to the edge routers' route-maps that strips all BGP communities from routes intended to be sent down to the spines, so a control flag can never again suppress the advertisement of the default route unintentionally. As the 2026 event showed, this protection is structurally limited: the well-known no-export community blocks advertisement before any outbound route-map runs, so a route arriving already flagged is never offered to the route-map at all.
• A route-map on the edge routers was configured to never set the no-export or no-advertise communities on a default route sent up to the reflectors or to other edge routers.

The assumption we drew, and why it was wrong

The 2024 event stemmed from our own tooling, triggered when an outside provider happened to flap its routes at the same time; the fix was within our own change process. From this, Teraswitch concluded that a falsely applied no-export would come from our own edge routers under maintenance, and that tooling and process corrections at the edges were sufficient protection. We never accounted for, or considered, the no-export community being added anywhere else: by a route reflector, or by an external ASN. Nor did we consider a default route arriving as "local" but not actually being usable as the readiness signal for the fabric spines. Those assumptions were incorrect. We remained vulnerable to the same flag arriving from any origin other than the one we had specifically resolved.

During the August 12, 2026 event the community was already present on the stale MIA1 route when it reached the affected edge routers. It was applied by our own global reflectors' direct-edge import policies. This bypassed all changes and improvements from the 2024 event.

The global fabric hardening deployed on August 12, 2026 closes this design flaw: the edge routers now force-advertise a locally generated default whenever they are genuinely able to carry Internet traffic, so the fabric always holds a valid default while its edges are healthy, regardless of where the edges learn their own default route or what control communities that route carries.

Resolution and recovery

Engineers identified the stale route within 10 minutes of onset. MIA1 was removed from the backbone by shutting down the loopback interface used for its SR-MPLS/IS-IS connectivity, and the AMS2 route reflector was powered off, halting further propagation and speeding recalculation. At that moment the responding engineer did not yet know whether MIA1 itself was the source or the AMS2 reflector was malfunctioning; what was known was that the affected sites had learned the route through AMS2's reflection, so both were removed. The team also halted route exports from the global route reflectors in the same window. The only operational change from that halt was peering traffic taking local and regional paths rather than crossing regions; customer and internal routes were maintained throughout by our Continuity Reflectors, which handle only internal and customer routes. With the route's origin now confirmed as er100a.mia1, removing MIA1 from the backbone withdrew the stale default at its source and restored service; powering off the AMS2 reflector removed the EU distribution path and forced recalculation. Reconvergence began at 04:16:15 UTC. The AMS2 reflector shutdown left a large backlog of route changes, so withdrawing and reprogramming the default route took time, and service returned progressively as each site reconverged on its local default routes. Full recovery, with routes recalculated and traffic patterns back to normal, took 33 minutes from the first alarm. MIA1 returned to the backbone through a maintenance shortly after the event.

Later the same day, working with our network hardware vendor, we deployed a global configuration change across all compute sites, including unaffected data centers, to improve the reliability of the connection between our Internet/backbone edge routers and our data center core networks. With this change in place, a stale or invalid route entering the fabric will no longer prevent traffic forwarding: sites will continue to forward via their local edge routers rather than losing reachability.

The rollout favored safety over speed. The configuration was updated on the edge routers feeding approximately 62 spine devices globally, one edge router at a time, completing before 15:00 UTC. At that point we did not yet know where the route had come from, so the global route reflectors were kept offline for the duration of the rollout to halt any propagation if it happened again, and the network ran on regional routes only.

How the fix works

The previous design had each edge router share the default route it had itself selected as the readiness signal toward the spines. The new design uses the BGP default-originate feature on the neighbor group from the edge routers toward the spines. The edge now generates the default route itself, so the route sent into vrf INTERNET always carries the local edge router as its next hop and can never inherit a next hop or attributes from a stale or invalid route elsewhere in the backbone.

Advertisement of this default is controlled by our automation system rather than by a route policy. When maintenance or a traffic drain requires an edge router to stop attracting traffic, the automation deploys a flag on that edge router that stops the advertisement, and removes the flag to restore it. A route-map on the session shapes the attributes of the originated default but does not control whether it is originated. This preserves the original design goal, that an edge router does not attract traffic from the spines before it is ready, retaining the blackhole protection the shared-default design was built for while removing the dependency on a route that transits the backbone.

Route reflector rebuild and software diversity

As further hardening, all backbone route reflectors were wiped and fully rebuilt with the latest operating system and reflector software in the week following the event. We also introduced reflector software diversity: a portion of the reflectors now runs a completely different software package, so a defect in any single implementation can no longer affect every reflector at once. This also hedges against the class of defect that lives in an interaction between specific software and firmware versions rather than within any single component.

Corrective actions

1. Deploy global fabric hardening: replace shared default route signaling with default-originate from the edge routers toward the spines, so a stale or invalid route entering the fabric cannot prevent traffic forwarding. (Complete, Aug 12, 2026)
2. Wipe and fully rebuild all backbone route reflectors with the latest operating system and reflector software, and introduce reflector software diversity by running a portion of the reflectors on a completely different software package. (Complete, Aug 12, 2026 and following days)
3. Reintegrate MIA1 into the backbone. (Complete, Post-event maintenance)
4. Correct the transposed community values and audit all route-maps for values colliding with control community families. (Complete, Week of Aug 10, 2026)
5. Increase logging and automatic debug state capture across backbone devices, including BMP for BGP monitoring, so forensic evidence is preserved without slowing service restoration. (In progress, Ongoing)
6. Deploy per-VRF default route monitoring: spine and leaf devices raise an alarm when a VRF that previously carried an active default route no longer has one. (In progress, Ongoing)
7. Move customer DoubleZero interconnects to within the data center fabric, so tunnel sessions no longer depend on paths outside it. (Planning, TBD)
8. Expand validation testing to cover alternative operating modes, including edge routers connected directly to the global reflectors. (Complete, Post-event)
9. Audit every edge router for leftover static default routes (none besides MIA1 were found), and update edge router configuration templates to disallow static default routes entirely. (Complete, Aug 12, 2026)
10. Block IS-IS from redistributing default routes and static routes. (Complete, Post-event)
11. Review and rebuild the BGP community scheme network-wide, an ongoing project that predates this event. (In progress, Ongoing)
12. Reclassify management network failover to out of band connectivity as a higher priority alarm than most other alerts, so it stands out during an alarm storm. (Planning, TBD)

Lessons learned

• Signal our own gear with a route we originate ourselves. When a default route is used to tell our downstream devices that an edge is ready, that route should be generated locally for that purpose, not borrowed from an external source that carries its own attributes and failure modes.
• The fate of routing between two devices should be decided between those two devices. A forwarding decision on a local link should not be inherited from a different site, a transit provider, or any other distant route source; the new design enforces this on the edge to spine boundary.
• Signals that look healthy can hide a broken plane. Traffic on shared links and fully functional management access both masked a failure confined to the INTERNET vrf. A failover whose alarms are buried in a larger alarm storm, as happened when the management firewalls moved to out of band connectivity, can easily go unnoticed during the response.
• Service restoration is priority #1 in these outages; engineers are only worried about restoring access. Evidence preservation must therefore be automatic: increased logging and automatic debug capture will preserve forensic state without slowing recovery.
• Fixing a root cause is not the same as fixing a failure mode. The 2024 remediation addressed the specific trigger (a tooling error) rather than the general class of failure (any untrustworthy default route reaching the fabric). The broader vulnerability remained until August 12, 2026.

Closing statement

This event was our mistake, even though triggering it required several small and unrelated conditions to align: a leftover route, a transposed digit, and a routine provider maintenance. We take pride in our choices and designs, and this event has driven us to reevaluate multiple other systems to verify their own integrity. The design carried an assumption about what could break: that any default route would be fully "cleaned" by its edge routers before it could affect the fabric behind them. This event proved otherwise: the no-export flag was honored rather than cleaned, withholding the fabric's default while every edge router operated normally. This assumption held for six years, but the 2024 FRA2 event was not properly considered beyond the specific triggers of that day. We are also not blind to what you are thinking after reading this: losing a default route inside our own fabric looks stupid. It is one of the most basic functions in networking, and it was not protected as well as it needed to be. If another provider did this, we would chuckle at it and be glad we had not made such a mistake. We learn from other providers' issues and successes all the time; perhaps someone will learn from ours.

At the same time, the surrounding design performed the way it was built to. Management access was retained to every device for the full duration, the offending route was identified within ten minutes, and affected sites recovered in roughly half an hour. This network absorbs failures every day, of equipment, routes, fiber optic spans, undersea links, and power, and customers rarely notice, because surviving those failures is where most of the design effort goes. This is the first event of this scale since the backbone took its current form in 2020, in a company operating since 2003.

We also understand the size and the sensitivity of what runs on this network. We remain unbiased about our customer base and about what customers choose to host with us, but we know exactly what earned that base and what set us apart from our competition: people trust us not to have failures. Simultaneous data center failures, in particular, are not acceptable to us. Our change control and management systems have always been built to make that outcome impossible, and every data center is designed to keep running through the loss of every other data center, in what we call island mode. In this event, several data centers all happily accepted the same bad route, and that has changed our view of how issues can propagate.

We would have preferred to find this failure mode in validation testing rather than in production, and that is the lasting change: the bar for validation and edge case testing has been greatly raised. The specific condition behind this event can no longer occur, the remaining hardening is being deployed in deliberately sequenced steps so that remediation never becomes its own event, and the network is stronger today than it was on August 11.

Brendan Mannella
Chief Executive Officer

Nick Zurku
Principal Network Architect

Appendix

A. Affected sites

• EU (LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2): Internet and inter-site backbone reachability lost
• APAC (SGP1, SGP2, TYO1, TYO2, TYO3): Internet and inter-site backbone reachability lost
• NA (MIA1): Temporarily removed from backbone as containment; operated in island mode with Internet routing over local transit only, later reintegrated via maintenance
• NA (All other NA sites): Not affected

B. Terminology

• default route (0.0.0.0/0): A catch-all route covering all Internet destinations. Teraswitch uses it internally to signal that an edge router is able to forward traffic to the Internet.
• vrf INTERNET: The routing instance on the spine layer that carries Internet routes for the data center fabric.
• route reflector: A router that redistributes BGP routes between sites so that every router does not need a direct session with every other router.
• IS-IS: The interior routing protocol Teraswitch uses to distribute reachability information inside its own network.
• out of band (OOB): Management connectivity that is fully independent of the production network, used to reach equipment even when the production network is impaired.
• next hop: The address a router must be able to reach to actually use a route. If the next hop cannot be resolved, the route is invalid and cannot be installed for forwarding.
• BGP attributes: Metadata carried with a route (communities, metric, AS path) used to identify its origin and control which route is preferred.
• no-export: A standard BGP community instructing routers not to advertise the route across a BGP boundary into another ASN.
• edge / spine / leaf: The three layers of a Teraswitch site: edge routers face the Internet and backbone, spines aggregate the site, and leaf switches connect customer servers.