Skip to Content

System Status

Partial Outage – Jax Cluster 2 Firewalls

June 22nd 2020

Status: RESOLVED

At approximately 10:00 AM Eastern, Cloudstar engineers noticed what was appearing as an intermittent connectivity interruption from Cogent Communications.

Two network engineers logged into the routing environment, while a third contacted Cogent Communications.

Upon further investigation, we discovered routing issues within the cluster 2 firewall environment.  One firewall had inbound only traffic, while the other firewall had no Internet traffic, but was able to pass Citrix data between clusters.

A remote restart was initialized on both devices as an onsite engineer was en-route to the datacenter.

Upon restart, the problem persisted.  Both firewalls were responsive to GUI and CLI inputs and indicated operational; however, asymmetric routing was still present on one firewall, while the other firewall could not pass any Internet traffic.

Engineers then discovered FortiOS system corruption on the first firewall.  This firewall was removed from the network, while the secondary firewall was manually configured as primary. Service was restored.

ROOT CAUSE: Cloudstar runs dual firewalls for the purposes of redundancy and to prevent downtime.  Redundancy is configured to be 100% automatic and hands off.  In this case, a partial operating system failure in the primary firewall prevented a clean failover to the backup firewall.  Once engineers identified the problem, the solution was to eliminate the bad firewall from the high-availability firewall cluster thereby restoring service.

 

0 Continue Reading →

Current Status

All Data Centers, Clusters, and Core Systems Online

 

PLANNED MAINTENANCE

Maintenance window:
Start Date and Time: 2020-May-07 18:00 UTC
End Date and Time: 2020-May-07 22:00 UTC
(Please note – for customers in USA: CST = UTC – 6hours, CDT = UTC – 5hours)

Action and Reason: Rerouting due to network capacity expansion.

Location of work: Palo Alto/ CA, US

0 Continue Reading →

Connectivity Issues – JAX3, Cluster 2

EMERGENCY MAINTENANCE is being performed on our JAX3, Cluster 2 environment.  

At approximately 11:47 PM EST, engineers received connectivity alerts from multiple systems.  A possible root cause was identified at 12:20 AM and a team of engineers are addressing the problem.

Sunday 3:52 PM ET: We expect systems to return online over the next several hours.  This page will be updated when any new information becomes available.

We thank you for your patience.

EMERGENCY MAINTENANCE COMPLETE

0 Continue Reading →

Cogent Outage – NJ

TICKET CLOSED – STATUS RESOLVED

3/21/20 12:07 AM – Cogent has repaired the fiber connection, service has been restored. Cloudstar engineers verified connectivity.

*As a reminder, Cloudstar services are still available via backup connectivity.  https://NJ1.Cloudstarsystems.com

3/20/20 9:24 PM EST Cogent Communications has acknowledged that they are aware of the problem and have isolated it to a fiber cut outside the NYC metro are.  Fiber technicians have been dispatched to the scene to repair the broken connection.

https://ecogent.cogentco.com/network-status

Customers in the Secaucus,NJ area are experiencing an outage at this time. Cogent is currently investigating. The Support master is HD11067006

3/20/20 8:42 PM EST  a Cogent internet outage is affecting connectivity at the NJ data center. Alternate ISP routes are available.

https://downdetector.com/status/cogent/   Down Detector shows the outage as emerging.

Alternate / Backup Route Login URL is:  https://NJ1.Cloudstarsystems.com

0 Continue Reading →