As a Cloudstar customer, you deserve honesty, transparency and information. At approximately 7:15 AM Eastern time, the Cloudstar Network Operations Center received multiple lost connectivity alerts. After briefly troubleshooting, it was determined that the cause was a local cable cut, to a fiber optic bundle servicing North Florida, located down the street from the affected data center.
Cogent Communications posted the following incident on their website opening master case number HD10626845.
It is noteworthy to mention, that prior to this cable cut, Cogent has provided 100% up-time service to the data center for approximately 8 years. Additionally, a single cable cut will not affect the circuit due to the ring topography. In this case, the fiber provider’s redundant ring failed.
WHAT WE DID
Cloudstar technicians fielded calls from customers (ranging from calm and understanding to extremely upset). Cloudstar engineers manually reconfigured several Citrix NetScaler Gateways to utilize a redundant blended circuit which restored services to most of the affected customers.
I THOUGHT WE HAD REDUNDANCY, WHY DID WE GO DOWN?
Before we can speak to redundancy, it is prudent to explain exactly what levels of redundancy are available, and what levels of redundancy can be expected as a Cloudstar customer. Redundancy comes in many forms; redundant servers, redundant storage, redundant power and redundant Internet.
Customers who purchased multi-data center redundant fail over, were NOT affected. It is worth noting that this is the norm in any hosting environment, wherever you shop for services. Typically, most customers do not purchase multi-data center hot-site fail over due to the high costs involved.
Customers who purchased redundant access links also, failed to notice any interruption of service.
Customers located in other server clusters, or at other data centers were not affected.
Of the customers who were affected, Cloudstar engineers manually re-configured access to provide a means to connect. Although not ideal, and not 100%, this allowed most affected clients to logon, access resources, and conduct business.
WE CAN NEVER GO DOWN, THIS IS UNACCEPTABLE
Cloudstar understands the importance of maintaining critical up-time, the urgency to which we must react during problems, and the financial impact and stress caused by the inability to access services. This causes a range of emotions for all who are affected; from upset, to angry, to irate. Unfortunately, no amount of time, money, or expertise absolves a customer from the potential of a systems outage. Nowhere is this more evident than the recent 4 hour long Facebook outage where users were calling 911 because Facebook was down, or when Chase experienced a national outage whereby all bank branches lost connectivity for over a day. The cases are endless, and it is important to understand that this is the world we live in, and it is our collective reality as consumers of technology services that no matter who we are, what we spend, or the decisions we make, every person and corporation is susceptible to potential periodic service interruptions.
Believe it or not, we’ve received telephone calls implying that this event was manufactured to cover up a problem the Cloudstar caused. To those individuals, I would be remiss if I didn’t mention that every provider from Amazon, to Google, to Microsoft, to Facebook and beyond is subject to the same realities. Yet these institutions are seldom questioned or asked for proof. In an effort to go above and beyond, Cloudstar drove to the site of the fiber cut and documented the incident with the following photographs.
Fiber Splicing Unit
Tire Marks from 18 Wheeler Running Over The Underground Access Box
(Located adjacent from BP Gas Station – Spring Park & University)
Fiber Optic Cable Is Round, and Contains Strands Of Glass Fiber.
(Note how the yellow strand was crushed by the vehicle).
I DON’T UNDERSTAND
Chances are, you’ve never given much thought to the millions of miles of fiber located just beneath the street. If you utilize the Internet, you’re traveling through optical fiber just like those pictured. The Internet services you rely on daily, are just as vulnerable. Cloudstar has options to assist in providing the level of redundancy our customers choose for their organizations.
Cloudstar will continue evaluating options for our our customers. For those that understand that these problems affect every hosting provider, every government entity, every bank, every hospital, and every individual, I sincerely appreciate your awareness as to the reality of of the world we live in. To those who are deeply upset, and “feel the need to take action”, I would ask you to consider the question, “Can we avoid the reality of how the Internet works”? Unfortunately, nobody is immune to a potential outage however there are steps that can be taken to decrease the odds. If you’re interested in learning more about hot-site fail-over options, redundant paths, and Cloudstar’s contingency plans, please let us know by requesting a consultation HERE.
Cloudstar would like to thank our customers for their business. We welcome your comments and feedback.
NOTE: Please note that services were restored at 12:45 PM Eastern.