Skip to Content

System Status

Warranty Replacement & Firmware Upgrades

TICKET CLOSED

Status Update:  Maintenance Completed Successfully

Ticket Number:  3829021
Location: Santa Clara, CA Data Center

 

Start Date: Saturday 11-7-20 2:00 pm Pacific Time
End Date: Saturday 11-7-20 6:00 pm – 8:00 pm Pacific Time
Work Window: 4-6 Hours

 

Status: Scheduled
Risk: Low
Maintenance Type: Warranty Drive Replacement & Emergency Controller Firmware Upgrade

 

Reason for Work: Storage Stabilization.

 

Description: Cloudstar engineers will be working hand-in-hand with Dell / EMC Mission Critical Pro Support to perform warranty drive replacements on a currently off-line storage array in addition to performing critical in-line updates.

Dell / EMC has identified several issues found within both Dell / EMC’s hardware and storage controller firmware that may lead to instability and will be performing an in-line warranty upgrade of both storage and storage controller firmware.

Best practices mandate minimal disk I/O so as to ensure a smooth upgrade. At the request of Dell / EMC Pro Support, customer virtual machines will be powered off during this process. The procedure will be carried out after verifying all customer backups are current and correct, and will be performed remotely by Dell / EMC Pro Support Engineers. Standby Engineers will be onsite, at the data center, with serial connections to the controllers, and remote access provided to Dell in the event a contingency restoration of controller firmware is required.

A post upgrade review of log files will be conducted by Dell / EMC Pro Support so as to ensure a successful in-line warranty upgrade at which time customer VM’s will be brought back online. Cloudstar customer services representatives will notify customers that it is then safe to return to normal operations.

Status updates will be posted online throughout the duration of the project as they become available.

NOTE: Cloudstar maintenance policies, parts availability, and end-of-month blackout procedures prohibit our engineering team from conducting this maintenance any sooner than it is currently scheduled.  Please be advised that there is no immediate threat to the environment; however, Dell / EMC advises that this maintenance be performed as soon as possible.  This equipment is currently under their management, and as the manufacturer, we defer to their professional guidance, expertise, and direction.

0 Continue Reading →

Telia Carrier – Planned Telco Circuit Maintenance

TICKET CLOSED

Cloudstar has received a planned work notification from Telia that will affect their circuits in California, including our Santa Clara, CA data canter.

The Internet backbone provider will begin performing a series of hardware upgrades to address both network capacity and expansions. Scheduled maintenance will be performed over 3 maintenance windows during off peak hours.

 

  • 10/26/2020 from 10pm – 10/27/2020 7am (Pacific)
  • 10/27/2020 from 10pm – 10/28/2020 7am (Pacific)
  • 10/28/2020 from 10pm – 10/29/2020 7am (Pacific)

 

While the maintenance windows are 9 hours long, it is unlikely that the resulting service interruption will last longer than one hour.

NOTE:  Cloudstar services will remain available via your alternative redundant access links.  Should you experience loss of connectivity during the maintenance window, it will only be temporary until the upgrade is completed by Telia or until you login via the redundant connection.

After each day’s work is complete, you will be able to resume normal use of the Telia links/URL’s.

Customers who have purchased their own dedicated circuits, use private circuits, or who utilize other shared circuits, will not be affected.

0 Continue Reading →

Partial Outage – Jax Cluster 2 Firewalls

June 22nd 2020

TICKET CLOSED

At approximately 10:00 AM Eastern, Cloudstar engineers noticed what was appearing as an intermittent connectivity interruption from Cogent Communications.

Two network engineers logged into the routing environment, while a third contacted Cogent Communications.

Upon further investigation, we discovered routing issues within the cluster 2 firewall environment.  One firewall had inbound only traffic, while the other firewall had no Internet traffic, but was able to pass Citrix data between clusters.

A remote restart was initialized on both devices as an onsite engineer was en-route to the datacenter.

Upon restart, the problem persisted.  Both firewalls were responsive to GUI and CLI inputs and indicated operational; however, asymmetric routing was still present on one firewall, while the other firewall could not pass any Internet traffic.

Engineers then discovered FortiOS system corruption on the first firewall.  This firewall was removed from the network, while the secondary firewall was manually configured as primary. Service was restored.

ROOT CAUSE: Cloudstar runs dual firewalls for the purposes of redundancy and to prevent downtime.  Redundancy is configured to be 100% automatic and hands off.  In this case, a partial operating system failure in the primary firewall prevented a clean failover to the backup firewall.  Once engineers identified the problem, the solution was to eliminate the bad firewall from the high-availability firewall cluster thereby restoring service.

 

0 Continue Reading →

Connectivity Issues – JAX3, Cluster 2

TICKET CLOSED

EMERGENCY MAINTENANCE is being performed on our JAX3, Cluster 2 environment.  

At approximately 11:47 PM EST, engineers received connectivity alerts from multiple systems.  A possible root cause was identified at 12:20 AM and a team of engineers are addressing the problem.

Sunday 3:52 PM ET: We expect systems to return online over the next several hours.  This page will be updated when any new information becomes available.

We thank you for your patience.

EMERGENCY MAINTENANCE COMPLETE

0 Continue Reading →

Cogent Outage – NJ

TICKET CLOSED

3/21/20 12:07 AM – Cogent has repaired the fiber connection, service has been restored. Cloudstar engineers verified connectivity.

*As a reminder, Cloudstar services are still available via backup connectivity.  https://NJ1.Cloudstarsystems.com

3/20/20 9:24 PM EST Cogent Communications has acknowledged that they are aware of the problem and have isolated it to a fiber cut outside the NYC metro are.  Fiber technicians have been dispatched to the scene to repair the broken connection.

https://ecogent.cogentco.com/network-status

Customers in the Secaucus,NJ area are experiencing an outage at this time. Cogent is currently investigating. The Support master ticket is HD11067006

3/20/20 8:42 PM EST  a Cogent internet outage is affecting connectivity at the NJ data center. Alternate ISP routes are available.

https://downdetector.com/status/cogent/   Down Detector shows the outage as emerging.

Alternate / Backup Route Login URL is:  https://NJ1.Cloudstarsystems.com

0 Continue Reading →