<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Hier der Abschlussbericht zum AMS-IX Ausfall. Kurzform: 100G-Ports zu Sub-Switch deprovisioniert aber in alter config (Peering-VLAN), Field-Engineer zu rasch unterwegs bzw. nicht vom NOC aufgehalten worden mit einem Loop.</div>
<div><br>
</div>
<div>500 von 600 sessions passt ca, wir hatten auch etwa die Quote an dropped sessions dort. </div>
<div><br>
</div>
<div>Cheers, /Rene</div>
<div><br>
</div>
<div>
<div style="font-family: Calibri; font-size: 11pt; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);">
<span style="font-weight: bold;">Von: </span>Konstantinos Koutalis <<a href="mailto:konstantinos.koutalis@ams-ix.net">konstantinos.koutalis@ams-ix.net</a>><br>
<span style="font-weight: bold;">Datum: </span>Wednesday 20 May 2015 23:37<br>
<span style="font-weight: bold;">An: </span>Tech-l mail list <<a href="mailto:tech-l@ams-ix.net">tech-l@ams-ix.net</a>><br>
<span style="font-weight: bold;">Betreff: </span>Outage post-mortem: 13 May 2015, 100GE loop on AMS-IX ISP Peering LAN.<br>
</div>
<div><br>
</div>
<div>
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">Dear Members and Customers<br class="">
<br class="">
As a follow up on the issue we had last week Wednesday May 13, 2015 at 12:20 we like to share what happened and what measures we have to taken to limit the possibility of a similar event happening in the future.<br class="">
<br class="">
The following is the sequence of events as we can recall them from different log files:<br class="">
<br class="">
- 00:00 - 01:00, all customers were moved away from stub-eq3-233, </div>
<div class=""> as announced in "#<span class="" style="font-family: 'Helvetica Neue';">174450,</span><span class="" style="font-family: 'Helvetica Neue';"> </span><span class="" style="font-family: 'Helvetica Neue';">Provisioning of new customer ports
at EQUINIX-AM3".</span><span class="" style="font-family: 'Helvetica Neue';"><br class="">
</span> The backbone links on the emptied PE were not disabled.<br class="">
- 10:39 - 11:00, 100GE modules were replaced on stub-eq3-233<br class="">
- 11:11 - 11:14, engineers started placing physical loops to<br class="">
test the newly installed modules & interfaces. Due to<br class="">
a miscommunication between the engineers on site and<br class="">
the NOC, the ports to be tested were still in the Peering<br class="">
LAN VPLS Instance.<br class="">
- 12:22, Interfaces were enabled using a script meant for testing new ports, </div>
<div class=""> and test traffic was generated. <br class="">
- 12:25, As 4* 100GE ports were looped and still in Peering LAN<br class="">
VPLS and no L2 ACL was in place, broadcast traffic over<br class="">
the loop makes customer router MAC addresses to show up<br class="">
behind the looped interfaces and attract traffic to<br class="">
these MACs. <br class="">
Linecards on all switches started having CPU spikes.<br class="">
Approx. 500 out of 600 BGP sessions went down on the<br class="">
Route Servers.<br class="">
- 12:29, NOC disabled the looped interfaces & the backbone links<br class="">
of stub-eq3-233.<br class="">
- 12:40, Approx. 500 BGP sessions on the route servers were back up<br class="">
<br class="">
<br class="">
Analysing the events has shown that there were two flaws in our <br class="">
procedures that made the outage possible.<br class="">
<br class="">
1: Blind reliability on the correct functioning of scripts to<br class="">
configure the switches into a state where this work could<br class="">
be executed.<br class="">
2: Miscommunication between the on site engineer and the NOC<br class="">
engineer overlooking the state of the platform. <br class="">
<br class="">
The measures we have taken define a more strict communication <br class="">
between the engineers doing on-site work and the NOC engineer<br class="">
looking at the state of the platform and the configuration changes<br class="">
necessary to execute the work. We also defined more clearly who <br class="">
is responsible for the configuration part while maintenance is <br class="">
going on.</div>
<br class="">
<div class="">
<div class=""><span class="">During the incident, the AMS-IX NOC being focused in ensuring that the stability of the platform was restored, </span></div>
<div class=""><span class="">neglected to send a notification to Tech-L on time informing all technical contacts about the outage. </span></div>
<div class=""><span class="">After reviewing our internal procedures we have updated them to ensure that, in any similar incident, </span></div>
<div class=""><span class="">the NOC engineer working with our on-site engineers will immediately notify all AMS-IX peering parties.</span></div>
<div class=""><span class=""><br class="">
</span></div>
<div class=""><span class="">Once again, we sincerely apologize for any inconvenience caused by that outage.</span></div>
<div class=""><span class=""><br class="">
</span></div>
</div>
<div apple-content-edited="true" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class="">Kind regards,</div>
<div class=""><br class="">
</div>
<div class="">Kostas (Konstantinos) Koutalis</div>
<div class=""><br class="">
</div>
<div class="">NOC Manager</div>
<div class="">Amsterdam Internet Exchange (AMS-IX)</div>
<div class=""><br>
</div>
<div class=""><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">Von: </span>Jürgen Jaritsch <<a href="mailto:jj@anexia.at">jj@anexia.at</a>><br>
<span style="font-weight:bold">Datum: </span>Wednesday 13 May 2015 21:12<br>
<span style="font-weight:bold">An: </span>Christoph Loibl <<a href="mailto:c@tix.at">c@tix.at</a>>, Jürgen Jaritsch <<a href="mailto:jj@anexia.at">jj@anexia.at</a>>, "<a href="mailto:klaus.darilion@nic.at">klaus.darilion@nic.at</a>" <<a href="mailto:klaus.darilion@nic.at">klaus.darilion@nic.at</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:atnog@mailing.atnog.at">atnog@mailing.atnog.at</a>" <<a href="mailto:atnog@mailing.atnog.at">atnog@mailing.atnog.at</a>><br>
<span style="font-weight:bold">Betreff: </span>Re: [atnog] AMSIX Heute<br>
</div>
<div><br>
</div>
<div>
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:11pt; color:black">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: black;">Servus die Runde,<br>
<br>
AMS-IX setzt doch genauso wie France-IX auf eine VPLS Implementierung? Da kommt man mit STP nicht weit.<br>
<br>
Unabhängig davon: führt solche Tests setzt man den Port in L3 mode und fertig ...<br>
<br>
Viele Grüße<br>
<br>
<br>
Jürgen Jaritsch<br>
Head of Network & Infrastructure<br>
<br>
ANEXIA Internetdienstleistungs GmbH<br>
<br>
Telefon: +43-5-0556-300<br>
Telefax: +43-5-0556-500<br>
<br>
E-Mail: <a href="mailto:jj@anexia.at">jj@anexia.at</a><br>
Web: <a href="http://www.anexia.at">http://www.anexia.at</a><br>
<br>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt<br>
Geschäftsführer: Alexander Windbichler<br>
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601<br>
<br>
<br>
<span style="color:black">-----Original Message----- <br>
<b>From:</b> Klaus Darilion [<a href="mailto:klaus.darilion@nic.at">klaus.darilion@nic.at</a>]<br>
<b>Received:</b> Mittwoch, 13 Mai 2015, 14:05<br>
<b>To:</b> Christoph Loibl [<a href="mailto:c@tix.at">c@tix.at</a>]; Jürgen Jaritsch [<a href="mailto:jj@anexia.at">jj@anexia.at</a>]<br>
<b>CC:</b> <a href="mailto:atnog@mailing.atnog.at">atnog@mailing.atnog.at</a> [<a href="mailto:atnog@mailing.atnog.at">atnog@mailing.atnog.at</a>]<br>
<b>Subject:</b> Re: [atnog] AMSIX Heute<br>
<br>
</span></span></div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
Am 13.05.2015 um 16:12 schrieb Christoph Loibl:<br>
> while testing one of the newly installed 100GE modules, accidentally placed a loop on the ISP peering VLAN<br>
<br>
Sollte STP so etwas nicht verhindern? Auf den Cisco und Juniper Switchen <br>
die ich kenne lässt sich zB STP gar nicht deaktivieren. Oder verwendet <br>
man auf Exchanges andere Switches ohne STP?<br>
<br>
lg<br>
Klaus<br>
</div>
</span></font></div>
</div>
</span>
</body>
</html>