————————————————————————————————————–

The Coreix tehnical team will be conducting maintenance on the backup network beginning at 23:00 (GMT+1) on Thursday 12th May 2011, lasting until 03:00AM (GMT+1) of the following morning.

Scope of work: Migration of backup1 and backup2 arrays.

Outage: Customers who take backup services on these two arrays (unmanaged NAS and RapidSync) will be unable to connect to the backup arrays and therefore any backups during this period will not run. Customers should re-run the backups once the arrays are on-line again.

Thank you for your understanding as we continue to improve our services.

If you have any further questions or would like any clarification please contact the support department via https://support.coreix.net/ or support@coreix.net

-Coreix Team

We are upgrading the EMC SAN to facilitate new features and increase our expansion options.

The maintenance will cause disruption to the services outlined below and for the scheduled maintenance window we request our customers refrain from logging onto the Emailplus Web Control Panel (cp.emailplus.net) or from using Outlook/OWA.

The following services will be affected:

PROVISIONING OF NEW RESOURCES OR MAILBOXES
SENDING OR RECEIVING MAIL VIA OUTLOOK, OUTLOOK WEB ACCESS OR BES/ACTIVESYNC MOBILE DEVICES
ACCESS TO PUBLIC FOLDERS

No mail will be lost during this period of maintenance, the mail will be stored on the exchange edge servers and upon normal service commencing, the mail stored will be delivered to customer mailboxes.

Thank you for your understanding as we continue to improve our services.

If you have any further questions or would like further clarification please contact the support department via https://support.coreix.net/

Coreix Exchange Engineering Team

For further information:
MINI SITE: http://www.coreix.net/newdc/
PHONE: 08000 226734
EMAIL: sales@coreix.net

Location: STA.LON2

Coreix :: Scheduled Network Maintenance – Saturday 25th December 2010

————————————————————————————————————–

The Coreix network team will be conducting core infrastructure network maintenance beginning at 00:00AM GMT on Saturday 25th December 2010, lasting until 02:00AM GMT of the same morning.

Scope of work: Upgrade firmware on Metro 5,6,7 and Metro 8

Outage: Customers who take a single transit feed from Coreix will notice an outage of approximately twenty minutes.

Equipment at risk: Transit customers on the Coreix Metro network.

Rationale: Coreix performs periodic Operating System upgrades on its equipment to improve security and features on its equipment.

Thank you for your understanding as we continue to improve our services.

If you have any further questions or would like any clarification please do not reply to this email but contact the support department via https://support.coreix.net/ or support@coreix.net

-Coreix Network Team

There was a failure in one of our network blades that affected a proportion of servers that where directly connected to this blade, our monitoring systems picked up the outage and an engineer replaced this blade within six minutes of the issue occurring. We apologise for any inconvenience caused

Location: STA.LON2

———————

The facility engineering team will be conducting power maintenance beginning at 02:00 GMT lasting until 06:00 on Sunday 20th Dec 2009.

Scope of work: Functional test of the new enhanced solution.

The commercial supply will be disconnected in a controlled manner by power engineers. This work will ensure the correct operation of the breakers in the Low Voltage equipment in switching the power source on to the Generators. A full risk assessment has been carried out and as an additional precaution; manufacturers will be onsite along with our data-centre engineers.

Outage: Non-service impacting activity, we do not anticipate any downtime during this testing.

Thank you for your understanding as we continue to improve our services.

_____________________________

STATUS UPDATE

We have successfully completed the scheduled maintenance.

———————

The facility engineering team will be conducting power maintenance beginning at 02:00 GMT lasting until 06:00 on Sunday 6th Dec 2009.

Scope of work: Functional test of the new equipment installed during the maintenance on the 4^th Dec 2009.

The commercial supply will be disconnected in a controlled manner by power engineers. This work will ensure the correct operation of the breakers in the Low Voltage equipment in switching the power source on to the Generators. A full risk assessment has been carried out and as an additional precaution; manufacturers will be onsite along with our data-centre engineers.

Outage: We do not anticipate any downtime during this testing.

Thank you for your understanding as we continue to improve our services.

_____________________________

STATUS UPDATE – 6th Dec 2009 04:19

The above mentioned maintenance activity planned for 6^th Dec 2009- 0200-0600 hrs GMT has now been postponed until further notice.

During our pre-change checks we identified a fault which could lead to a degraded service on the cooling system and further investigation with the relevant engineering staff established that given a certain sequence of events full cooling to the data halls during the maintenance could be affected.

After a full risk assessment it was agreed that the increased risk was unacceptable and led to the decision that the change should not go ahead.

We apologise for cancellation of this change and will attempt to rearrange it as soon as possible when the risk profile can be reduced to an acceptable level and we will endeavour to give as much notice as possible for the future scheduling of this change.

———————

The facility engineering team will be conducting power maintenance beginning at 03:00AM GMT on Friday 4th Dec 2009

Scope of work: Testing LV switch gear and failover from Primary to Backup power.
Outage: We do not anticipate any downtime during this testing.

Thank you for your understanding as we continue to improve our services.

_____________________________

STATUS UPDATE – 4th Dec 2009 08:00

We have successfully completed the scheduled maintenance. There will be further non-intrusive maintenance, which is currently in the planning stage; this will fully test new components installed last night. We are planning to get another change window this weekend.

Why did this incident occur?

This issue occurred due a failure of the switch gear that allows us to move between the power grid and backup power sources. This problem caused the site to not return to grid power once it became available. You can view an excerpt from the TATA’s preliminary report at http://status.coreix.net/?p=15 which gives a full report on how the facility handled this incident and currently proposed solutions to ensure this incident is not repeated.

What is Coreix’s major incident procedure?

At the current time our major incident procedure for a power outage is as follows:

• Staffing procedures:
  • Contact all staff not currently on shift to arrange for them to immediately head to the datacentre.
  • All technically capable staff located at other sites to be diverted to the datacentre.
  • One staff member to be designated to deal with ticket management, external monitoring of progress and updating of status page.
• Deploy an automated telephone message on all phone systems pointing clients to http://status.coreix.net.
• Once service has been restored:
  • Network engineers to ensure that all core services come on line while the technical staff ensures that all racks are powered up.
  • Staff to be split half and half between responding to all technical issues within tickets and bringing servers online based on monitoring alerts.
• Once all monitoring alerts and support tickets have been dealt with:
  • Restore telephone service.
  • Start investigation into root cause of issue.

While we understand that customers would like to be able to contact us by telephone in this situation we feel this is not viable as it would double, if not quadruple the time taken for us to restore service to all clients.

What happened on the day?

On the day we were able to contact the majority of our staff and had a double compliment of staff on site before the power was resumed to the site. The phased start-up proceeded according to plan, with core services being returned within 10 minutes of power on and all racks powered up within 15 minutes of power on.

Due to a DNS failover error our status page was not available and we were unable to resolve this issue until power was returned to the site; at this point we manually pointed the DNS to our offsite backup.

Once power was resumed, all mail sent during this period spooled into our support desk. A mass e-mail was sent to all open tickets to confirm power had been restored which eliminated clients that had returned to full service. This procedure allowed us to concentrate solely on clients with continuing issues.

Our first priority thereafter was getting customers back online whose servers had not powered up cleanly after the hard reboot. As customer uptime is a core principle of our service we worked non stop until we were happy that all clients where back online, at this time we enabled our phone system and moved back to providing our standard service.

What has Coreix learnt from this incident?

While we feel that the technical element of our response ran as efficiently as possible there are several areas where we can see room for improvement, mainly in relation to customer contact during this period:

• Status Page

This we feel was our major failing; our status page was set up to failover from an onsite location to an offsite location in the event of a power or network outage. This process did not work as expected and the issue was compounded by the fact that we could not update the DNS to manually point to the offsite location until our primary DNS came on line. This was due to the fact that all records replicate out from this server. We have already resolved this for future incidents by permanently running this service offsite.

• Ticket Management

While we were able to rapidly deal with all technical issues we found limitations in utilising the ticket system to prioritise issues when we had a couple of hundred issues open at once. This resulted in there being no response to tickets until a ticket saying the issue had been resolved. We understand that this is not ideal for our clients and we are currently trailing alternative systems to allow us to queue tickets in a manner that will allow us to provide more feedback.

The future

With regards to the site itself while we are concerned in regards to this incident and will be further investigating it to ensure that this issue does not occur again. We feel that we must point out that this site has been extremely reliable in the past and is the first power incident in our five years here. The site held load perfectly through at least 6 or 7 other local power issues in this time and is in fact one of the few sites in London that hasn’t had any power issues until now.

Closing Statement

On behalf of Coreix, I sincerely apologise for the disruptions to your business and for the lack of information provided during this trying time. We fully understand that these failures negatively impact the businesses of our customers. Coreix has been working hard since this incident occurred to ensure that in any subsequent event we will be able to provide improved communication channels to allow all clients to receive regular updates without slowing the technical resolution down.

If you would like to talk to us further about this incident please contact info@coreix.net and we will be happy to assist you. If you would additionally like us to keep you informed regarding the changes to procedures made by both TATA and Coreix in relation to this incident please contact the Operations Director, Shazad Boota via s.boota@coreix.net.

As always, your feedback is welcome.  Please be honest with us about your expectations and how we can improve our service.  We will do our best to restore your trust in us and I want to thank you on behalf of Coreix for standing by us during this difficult period. I’d also like to extend our thanks to our technical team for their tireless efforts to deliver our service and support our clients over this period.

Shazad Boota
Operations Director, Coreix Limited