admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
9pm Friday July 9 CST - 6am CST Saturday July 10
Data center is moving to better location.
Maintenance Window Notification: This e-mail serves notice that your server(s) are scheduled to be relocated between: 9pm Friday July 9 CST - 6am CST
Saturday July 10.
Forum, GSB cloud and trademaid.info will be down.
|
|
|
admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
As a valued customer, Oplink.net is committed to keeping you informed about any changes in the status of your service with us. This email is to alert
you regarding a maintenance which will affect ALL of our services:
Start time: 10:00pm Central Time 8/24/2019
End time: 10:30pm Central Time 8/24/2019
Expected Outage/Downtime: 30min
Our upgrade did not complete during the time window from 12-2am. We need to complete one final step in our 100G upgrade. We hate doing this so soon,
but want to avoid maintenance events on weekdays. Your understanding is appreciated.
We appreciate your patience during this upgrade.
Thank you for your business.
|
|
|
admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
Data center is moving to better location.
Maintenance Window Notification: This e-mail serves notice that your server(s) are scheduled to be relocated between: 9pm Friday July 9 CST - 6am CST
Saturday July 10.
Forum, GSB cloud and trademaid.info will be down
|
|
|
admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
Just a reminder to all
Data center is moving to better location.
Maintenance Window Notification: This e-mail serves notice that your server(s) are scheduled to be relocated between: 9pm Friday July 9 CST - 6am CST
Saturday July 10.
Forum, GSB cloud and trademaid.info will be down
Thanks received (1):
+1 Carl at 2021-07-09 08:46:44
|
|
|
admin
Super Administrator
Posts: 5069
Registered: 7-4-2017
Member Is Offline
Mood: No Mood
|
|
If this outage was not fixed my this morning, Australian time I would have rebuilt the GSB server in the data center I use for my trading / VM
service.
here is the details. Apologies to all.
This implies there is going to be a shorter outage when the data center switches fibre cables again
8/7/21 Network Outage Explanation
We apologize to all of our customers for this 27-hour outage. We know some of you are receiving major complaints from your businesses and clients.
Please feel free to share this email with them, to help them understand what happened.
What caused this Outage?
At 6:10am CDT August 7, a major fiber optic connection between Level3 and our new Datacenter went off-line. The sole purpose of this fiber is a
point-to-point link between the new and old datacenters to keep both online while we accomplish the large task of moving. Please understand this
temporary link was needed for only 2 months, and will be permanently replaced with our 100 Gbps LUMEN(Level3) uplink. Unfortunately, disaster struck
the point-to-point fiber line and the new datacenter became isolated.
We notified the loop provider around 6:20am. They found a major cut#1 about 1,000 ft. from the LUMEN end of the loop. Cable was lying severed on the
ground in the grass and details of how it got there are not clear. We think a tree fell on the line dropping it to the ground where it was cut by a
lawnmower. Crews worked through the night to replace 300 ft. of cable and to splice approximately 150 fiber lines on each end.
About 12.45 pm CDT, after checking for light transmission continuity, a second cut#2 was discovered 3-4 miles north of cut#1. The crew says this
location was damaged by thieves attempting to steal copper, a very rare occurrence. The two coincidental cable breaks look to be unrelated based on
damage characteristics. A second crew worked through the night to reconnect cut#2.
Simultaneously fixing two major breaks complicated the repair timeframe. Because of the total scope of damage, the ETR, (estimated time of repair) was
difficult to determine. The unprecedented situation of a double fiber cut caused the outage to last so long.
If Oplink.net was multi-homed could this outage been avoided?
Being multi-homed would have been first established at the LUMEN datacenter and all traffic would still be carried on the temporary link during our
move. If the temporary link between the old/new datacenter had additional redundancy, we would not have gone down, but the cost to establish a second
link is high, requiring a minimum 12-month term, and our business judgment was to forego a second link.
Our core network to the Internet is not presently multi-homed. We use Level3/LUMEN bandwidth because they have arguably the best tier1 bandwidth with
many protections and fault tolerances built in. Down-time has been very rare.
How do you plan to avoid a repeat of this outage?
On 8/19/2021 our Level3/LUMEN 100 Gbps uplink is scheduled to be permanently relocated to our new datacenter facility and connected by means of a
LUMEN underground fiber ring with dual redundant cables. The temporary link will be discarded and our vulnerability to damage of exposed cables will
be removed. We will send a maintenance email later this week to confirm the exact day and time. We may push this timeline up. We are reviewing all
options.
The LUMEN network design/configuration has worked and allows us to focus on affordable high bandwidth options to our customers. However, in addition
to our redundant Level3 100 Gbps uplink at the new datacenter, we are considering to add another provider such as Cogent or Hurricane Electric to
become multi-homed. Please note, this would not have prevented the outage this time, but will ensure we do not have another outage if the LUMEN uplink
goes down.
Thank all of you for your business
Our purpose is to provide the best service possible to our customers and this outage is devastating to us. We are proud of our customers and we
pledge to do all we can to avoid anything like this from happening again.
Sincerely,
Oplink.net Staff
281-445-9800
|
|
|
![[*]](images/davis/default_icon.gif){kind=icon}
