Incident - Gosport X9 - Server offline #22

New Issue

Brandon Currell · 2024-11-27T15:32:01Z

Brandon Currell commented

2024-11-27 15:32:01 +00:00

We are aware of an issue with Gosport X9 resulting in the server being offline. This is being investigated. Affected customers will be contacted shortly.

Brandon Currell added the

Network Fabric Router

Datacentre

labels 2024-11-27 15:32:09 +00:00

Brandon Currell pinned this 2024-11-27 15:32:11 +00:00

Brandon Currell commented

2024-11-27 17:00:40 +00:00

This issue has been confirmed as an interface drop. Tickets are open with the datacentre partner to find out what happened.

Brandon Currell commented

2024-11-27 17:58:06 +00:00

Server has been back online since 15:29 and seems to be issue-free. Awaiting confirmation of the root cause.

Brandon Currell commented

2024-11-28 08:55:49 +00:00

Interface drop was caused by a datacentre engineer disconnecting the wrong cable during a scheduled change.

There was an open low-risk change ticket open for Gosport X9, where a cable going to a firewall was being disconnected so that it could be patched through a switch. Explicit instructions were provided about what cable to remove, however due to an internal labeling issue with the datacentre documentation, the wrong cable was removed.

The issue was resolved without intervention from us: they traced the correct cables and completed the change as requested, restoring connectivity in the process.

The datacentres internal documentation has been updated to reflect the correct labelling.

Due to the resilient nature of our connections, there was no extended customer impact. Customer internet breakout traffic had already been routed away from this server as part of the change ticket, so the only impact was a blip due to the links being forcefully interrupted, until BGP kicked in and the paths were updated.

This incident is now resolved.

Interface drop was caused by a datacentre engineer disconnecting the wrong cable during a scheduled change. There was an open low-risk change ticket open for Gosport X9, where a cable going to a firewall was being disconnected so that it could be patched through a switch. Explicit instructions were provided about what cable to remove, however due to an internal labeling issue with the datacentre documentation, the wrong cable was removed. The issue was resolved without intervention from us: they traced the correct cables and completed the change as requested, restoring connectivity in the process. The datacentres internal documentation has been updated to reflect the correct labelling. Due to the resilient nature of our connections, there was no extended customer impact. Customer internet breakout traffic had already been routed away from this server as part of the change ticket, so the only impact was a blip due to the links being forcefully interrupted, until BGP kicked in and the paths were updated. This incident is now resolved.

Brandon Currell closed this issue

2024-11-28 08:55:49 +00:00

Brandon Currell unpinned this 2024-12-06 10:51:27 +00:00

Sign in to join this conversation.