(sau titlul alternativ)

Ghidul serviciilor de hosting profesionist – asa se face !

Am avut ocazia (daca pot sa zic asa) sa observ live cum se trateaza o situatie de criza la un hosting profesionist. Sa incepem cu inceputul. Acest site este hostat la imountain.com, acelasi hosting care ofera serviciile si blogului lui Zoso si altor site-uri de succes (vezi Cocalari, Pitzipoance). Hostingul este foarte ok pentru pretul practicat, singurul inconvenient pentru romani este ca serverele sunt in US si pagina se incarca de 5 ori (credeti-ma, am testat) mai incet decat daca s-ar fi aflat pe un server metropolitan. Insa pretul si calitatea serviciilor nu se compara cu a niciunui provider de pe la noi. In continuare o sa exemplific acest lucru.

Duminica (01.06.2008) a avut loc un incident. Nu este vorba de alegerile locale sau de Inima de Tigan A avul loc un incident in clusterul care tine bazele de date imountain din cauza unui update gone wrong. Initial nu am fost afectat deoarece dupa primul mesaj pagina mea inca se incarca si era bine-mersi :

Dear Clients

We have an issue with the mySQL cluster.

We are looking into the cause and fix at this moment. We will update you as soon as we find the solution.

We apologize for the inconvenience.

Support
iMountain.com

Asadar am trecut repede peste acest moment fara a-i da prea mare importanta, insa in urmatoarele ore se va produce situatia de criza despre care aminteam in titlu si care va cauza caderea completa a paginilor web care erau hostate si aveau nevoie de mysql pentru a functiona :

Dear Clients

Some of the databases on this server were corrupted. It is pretty rare but it happens once in a long time.

We are identifying the corrupted databases and restoring them.

ETA for a fix is 60 minutes.

Thank you for your understanding.

Support
iMountain.com

———————————————————————-

mysql.imountian.com down again

Dear clients

One of the servers has developed a problem in the cluster. We are dropping it from the cluster and rerouting the traffic.

This is a carry over from the problems we had last night.

When it rains it pours.

We understand you’re anxious to get your site up and we’re working as quickly as we can.

ETA for a fix should be about 30 minutes. It will take a few minutes longer after that for DNS to update to the new cluster

Support
iMountain.com

———————————————————————-

mySQL.imountain.com delayed

Dear clients

RE: mysql.imountain.com

This is taking longer than expected. We have asked some of our day shift to come in early to assist.

We’re getting this back up as quickly as possible.

Again we know apologies don’t help you but know that we do regret this incident and are doing all we can to resolve it quickly.

If you are on our other clusters, you are not affected by this outtage on (mysql.imountain.com)

Support
iMountain.com

In acest moment toata lumea care avea paginile hostate la imountain era probabil panicata deoarece serviciul era down de cateva ore bune – de fapt cam toata ziua-lumina de duminica a fost compromisa. Mesajul prin care se comunica remedierea problemei si urmeaza scuzele de rigoare :

mySQL.imountain.com update

Dear Clients

The database restore has been completed on almost all databases at this time.

If yours is not up yet, it should be done within the next few minutes.

We have most of our crew in now so for the next couple of hours, we will be working on a resolution to avoid what happened to the cluster. Our ETA’s were also way off and we will also work on that.

Again, due to the number of emails we received, we can’t answer each one indvidually. If there is an emergency, please send it again.

For the downtime credit please wait until tomorrow (Monday) to send an email in to billing(at_)imountain.com and request your 1 week downtime credit.

Again, thank you for your patience and we apologize for the delays.

Support
iMountain.com

———————————————————————-

Dear Clients

If you are on mysql.imountain.com, this is an update on.

The database restore is almost completed and should be coming up shortly.

We will be providing a 1 week service credit for those affected. If you wish to have this credit applied. Please email us tomorrow to our billing email address “billing(at_)imountain.com”

Thank you for putting up with the outtage. It is unfortunate and embarrassing for us and we strive to not have it happen again.

We will be calling an emergency meeting today. Please limit your support tickets for the next few hours so that we may have some time to discuss improvements to our cluster service.

Our apologies again, your database should be up shortly.

Thankyou
Support
iMountain.com

———————————————————————-

Dear Clients

Our first fix did not work, the data re-corrupted.

We’re dropping the replication and doing a restore from backups.

Our ETA is now 60 minutes for the restore to complete.

We have received several emails on the ETA and we apologize if we cannot answer them all. Please bear with us for a bit longer.

Support
iMountain.com

Adica au cam dat-o in bara, au cam pierdut datele din ultimele ore si asta s-a lasat cu niste reduceri si bagarea staff-ului intr-o sedinta. La cati bani dau pe hosting (5$ 11$) nu prea ma interesa respectiva reducere si totusi am vrut sa vad cat de operativi sunt baietii:

Greetings Sir

We will be crediting your account this week.

Thank you for your understanding.

Billing
iMountain.com

Ok – in acest moment probabil ca ar fi ajuns oricare companie de hosting. Totusi comunicarea a jucat un rol crucial. Desi au ratat termenele initale este foarte important sa mentii clientul “in the loop”, informat asupra crizei. Astazi insa primesc inca un mail care ma notifica de masurile luate in sedinta de dupa criza :

Dear Clients

As a result of our discussions from the crash of the mysql.imountain.com cluster, we are implementing a few new backup procedures.

This was a fluke of a crash because we did an update to the mySQL cluster and it looked fine at the time but didn’t develop issues until several hours later.

So for future operations:

1) Prior to doing updates, we will always do a backup instead of relying on our early morning backups.
2) We will have a live server running a copy of the database at all times. This will be an unattached server which will have the most recent backup of the database on-line.

Fix #1 will ensure that if we do an update that goes badly, we’ll have a fresh copy of the database to restore, not one that is up to 24 hours old.

Fix #2 will provide a server we can switch you to in case #1 isn’t available. This will eliminate the long wait it takes for mysql databases to restore.

Database replication which we use is usually extremely reliable and it’s what keeps things running smoothly even if 1 server fails. In this circumstance, 1 problem was replicated across all the servers so that just broke everything.

We feel these changes will provide a quality hosting environemnt.

Thank you for your patience through this ordeal.

Support
iMountain.com

Mie unul mi-a placut cum am fost tratat si informat asupra situatiei. Multi nu ar fi considerat necesar sa trimita ultimul mail care clarifica situatia si explica ce masuri s-au luat pentru a nu mai ajunge iar la acest tip de criza. Asta numesc eu calitatea serviciilor de hosting si transparenta totala a provider-ului ! Situatiile de criza vor exista intotdeauna, important este cum le tratezi si cum pastrezi clientii in timpul crizei.