« Currently on my mind: Webfusion server downtime and SLA's… »

November

If you are an existing Pipex / Webfusion customer and are experiencing repeated problem with service levels, support and downtime for your web site(s) I strongly urge you to report them to Trading Standards and to the BBC's Watchdog. If enough of us make an issue, perhaps something will be done!

A reply from Pipex

Hi Paul

Apologies for the delay in responding.

Regarding the mango server failure, I wish to assure you that everything was done as fast as possible. Contrary to your comparisons of a big company being hosted, I am sure the amount of customers on this one server amount to far more annual fees than any one customer, however as I say everything was actioned with complete priority.

The server itself experienced complete hardware failure, and we had another server waiting ready for the back ups to be applied to. The problem was with the restore from the back ups, in that the initial process took far longer than normal to restore to the new server. A decision had to be made whether to interupt that process or let it run, taking into account that if we interupted it, we may have to start again from scratch and wait just as long again, if not longer. The process however did complete eventually, and it was then that we found some of the restored files needed attention, confirming thgat some of the back process had been corrupt for the Mango server. The problem with those files and the restore was related, and as you have seen had a knock on effect, which took further time to resolve.

A system is being established this week, that will see other servers mirroring these shared servers so that in the even one server had hardware failure, the downtime and interuption will be minimal. This mirrored server wil only be used in the case of hardware failure, but will see that standard restore to online time, reduced from the usual 10 hours to about 30 minutes. Of course moving forwards Mango is on brand new hardware, and should not require such systems, but they will be there if needed.

Additionally, we may have discussed the companiess longer term plans, in that we have been developing a new platform for over a year now, that will see a new shared hosting platform put in place, that will provide load balancing acroos all of the service, and see even minor interuptions covered by a similar mirrored effect. Of course for such a large company to make such a big change, this development takes time, and testing. We hope to have a date for this projects release in the new year.

As you can see much is being done, and the management are well aware of the situation, and this failure certainly served to hilight immedaite action to be necessary. However that does not change what has happened, and we are not applying our normal terms and conditions in this particular instance. I am able to offer one months free hosting, by form of an extension to your account on the Mango server, and hope that in that month we may have the opportunity to restore some lost confidence.

Again please accept my sincere apologies, and I will apply the extension on receipt of your confirmation.

I look forward to hearing from you Paul.

Regards

Andrew Tellwright
Customer Service Team Leader
PIPEX

0115 9170000

To which I had to bite my tongue before replying:

Dear Andrew,

It is with bitter irony that I accept your offer, the irony being that even today the Barium server ran apparently ran into problems - again - and I was queueing for nearly 20 minutes to support before being informed that it needed a reboot and would be back up in about ten more minutes.

Even now, a little after midnight, all the domains on Mango are running so slow as to be unusable. And no, it isn't me, as Barium, Uranium and Neon are all fine. (12:23am, Mango is back up to speed).

Have to say, not doing a lot for my confidence, hmmm.

~ Paul Ackerley

The load balancing promise is nice (Err, why isn't it there already?), but the fact the backups were corrupt makes me nervous. Plus, as I've said, to me there seems to be an underlying attitude that the customer is always wrong and needs to be kept in the dark for their own good, like the naughty children we are. Really, in my opinion, compared to virtually all North American ISP's, they - like most in the UK - are grossly overpriced and under specified. Not happy, but for peace of mind and someone local to scream at, I can deal. But when I'm seeing almost daily downtime and problems, well, time to start looking for a new home, hmmm.

That's it, I've have it with Webfusion!

1pm, 23rd November, 2005.
Ackadia is down, YET AGAIN. I'm starting to take this personal. As far as I'm concerned Pipex's claims of 99.9% uptime are a joke! I'm not putting up with this


After queuing on the phone for a good 15 minutes I eventually reached a technician. Barium, my server, has crashed - again - due to load issues, apparently… Awaiting the reboot.

Want my opinion? If you are thinking of hosting with Pipex, Webfusion or it's sister companies, don't!!! The downtime is getting far beyond a joke and what I think of customer services can't even me worded on a family website. I'm starting to get the feeling that if we (users) aren't aware of the downtime, well, it didn't happen. What is wrong with a courtesy email apologising for downtimes as and when they happen, eh!? Now that is customer service. Proactive firefighting. Keeping quiet on the subject smacks of a cover up, for me! To take a line from Monty Python:

Trying to reach customer support, but the lines are continually busy. Guess I'm not the only one ticked off about now, eh! Got through, waiting an email back. More to the point, reminded them that I'm waiting for an email back and added my aggravation over continual downtime on the main server to the list.

1.33pm, Ackadia is back online.
We can add at least another 40 minutes to offline time just for Barium, for my site, and for the hundreds, even thousands of other domains hosted on the server. *Mutter*

WoHoo. Added Frappr to the site!

If you are anything like me, you'll love this!

Frappr!
Why not place yourself on the Ackadian world map. Feel free to leave your picture and comments.

Waiting a response from Webfusion

Just fired off that email to Pipex's customer services:

17th November

Spoke briefly with Andrew T. ( hosting-customerservices@pipex.net ) about making a claim for losses and he will look into it considering the extraordinary situation with the Mango server.

Now to compose a polite but firm email explaining that, no, honestly, I'm REALLY, REALLY wound up over it all!

15th November

The damned thing is down, AGAIN:

Tuesday, November 15 2005 @ 11:55 AM
Contributed by: Support

The mango server is currently performing a standard scan disc following a reboot. This is a normal procedure and takes place after a number of reboots.

The server should be back online shortly.

Thank you for your patience.

Straight off I've ordered my card details taken from their system and I WILL be moving to another provider. As it stands I have paid for a years reseller hosting, up front. The fact that the all my domains have been and still are offline is immaterial. I can cancel my account any time I want - but they keep the money, thank you very much…

There's that many trying I can't get through to customer services ( 0115 917 0000 - option 6 ) so I'm spending - I'm wasting - the morning reading Pipex/Webfusion's convoluted Terms & Conditions before I take the matter to Trading Standards and seek legal advice later today.

Annotated Extracts from the T&C:

3.2 PIPEX will use its reasonable endeavours to provide the Services in accordance with any timescale set out on the Order Confirmation(s), but will not be liable to the Client where, using those endeavours, it fails to meet any timescale.


My contemptuous translation :
If we screw up, tough, you have agreed we are not liable for our own mistakes.


3.8 Without prejudice to its other rights and remedies, PIPEX may at its sole discretion suspend the provision of the whole or any part of the Services (temporarily or permanently) and will have no liability to provide the Services on the occurrence of any of the following events:
3.8.1 notified or unscheduled upgrade or maintenance of PIPEX's IT systems;


My acrimonious translation :
No really, you should have read the small print, if WE mess up, tough, we have you by the short and


4.1 The Client acknowledges that, given the nature of such services, PIPEX cannot guarantee that the Services, when delivered via the Internet, will be uninterrupted or error free.

4.2 To the fullest extent permitted by law and save as provided elsewhere in the Agreement, the Services and any Client Systems and Ancillary Systems are provided by PIPEX to the Client on an "as is" and "as available" basis and no warranty or representation (express or implied) of any kind are given in connection with the Agreement including as to satisfactory quality and fitness for a particular purpose. In particular, PIPEX gives no warranty or representation that:

4.2.1 the Services will meet the Client's requirements;

4.2.2 the Services will be provided on an uninterrupted, timely, secure or error-free basis; or

4.2.3 any results obtained from use of the Services will be accurate, complete or current.

4.3 PIPEX warrants that it will provide the Services with reasonable care and skill and in accordance with any SLA. PIPEX will not be liable for a breach of such warranty unless the Client notifies PIPEX in writing of such failure within 14 days of the Client becoming aware of the failure.

4.4 If the Client makes a valid claim against PIPEX based on a failure by PIPEX to comply with the warranty set out in clause 4.3 PIPEX may, at its option, take such steps as it deems necessary to remedy such failure or refund such part of the Fees as relates to such Services, provided that the liability of PIPEX under such warranty will in no event exceed one and a quarter times the amount of the Fees paid to PIPEX by the Client (excluding VAT and expenses) in the 12 month period prior to the date on which the Client makes the claim. If PIPEX complies with this clause, it will have no further liability for a breach of the said warranty.


My sarcastic translation :
Our lawyers clever wording really have this sewn up, eh. So sue us, it's your money you'll be throwing away! As we said, you really should have read the small print before you agreed to pay us for services we don't really have to supply if we don't want too…


5 Client's Obligations
5.1.3 not use the Services, Ancillary Systems and/or Client System or allow them to be used for the publication, linking to, issue or display of any material which in the absolute discretion of PIPEX may harm PIPEX or any of its Associated Companies or clients or bring PIPEX into disrepute or which calls into question any action taken by PIPEX on the Client's behalf;


My contemptuous translation :
Okay, I might have to move this Blog! Apparently, my agreement includes a binding clause that says, if - when - we take all your resold websites and domains offline you cannot tell anyone - that might affect our sales. We want more fools like you to pay up front without having a solicitor go through the agreement with a fine toothed comb.


6 Payment Terms 6.1 The Fees are payable to PIPEX subject to the following conditions: 6.1.1 Fees payable monthly or yearly will be paid in advance and will not be refundable in whole or part if the Agreement or relevant part is terminated during the period to which the payment relates;


Translation :
Like we said, if we let you down so badly you feel the need to move to another host, that's fine, we have the money up front!
I so very much wish I'd paid on a monthly basic, I tell you!


9.3.2 for any other kind of loss, one and a quarter times the amount of sums paid by the Client to PIPEX pursuant to the Agreement (excluding VAT and expenses) during the preceding 12 month period.


Hmm, I wonder if that includes the inconvenience and losses due to weeks of unscheduled downtime?


9.4 PIPEX will not be liable to the Client in contract, tort, misrepresentation or otherwise (including negligence), for any indirect or consequential loss or damage, costs, expenses or other claims for consequential compensation whatsoever, or for any loss of profit, loss of business, loss of contract, depletion of goodwill or otherwise (whether direct or indirect), and whether or not caused by the negligence of PIPEX or its employees, agents or authorised representatives, which arises out of or in connection with the Agreement.


My shocked translation
Ewwww, that's evil.
If we are negligent, you agree to suck it in and deal with it, dude!


PART 3 - SUPPORT SERVICES
26.4 PIPEX will use its reasonable endeavours to provide the Support Services in accordance with the SLA.

26.7 If a Defect occurs, the following procedure will be followed:

26.7.2 PIPEX will analyse the Defect and use its reasonable endeavours to rectify the Defect in question or propose a solution in connection with the same, within ten Business Days of being notified of the same under clause 26.6.

Final thoughts:
Really must look up their Service Level Agreement.
Got to love the caveat about working days. I can just imagine them telling DSG, sorry dude, we don't do weekends. Quite handy that the problems started on a Thursday, eh. At it's worst (allowing for bank holidays), you website/server has to be offline for up to 17 days before they agree to "propose a solution". Ah, pardon me, they won't tell you if there if an error… So, that's (up to) 17 days after you found out the hard way by someone complaining the website down that they are at liberty to agree maybe they should do something…

*Mutter*
Have these suits never heard of the good English guidelines? What's wrong with putting it all into simple terms for the layman, eh!?
For obvious legal reasons I will clarify that the above sentiments represent my frustrated and exasperated feeling at the way the situation was handled in this one instance and in no ways questions the effort they may have put into rectifying the problems.
Nevertheless, given this isn't my first issue with the way Webfusion has been managed since being taken over by Pipex so I firmly stand by my extremely negative feelings about the company as a whole and would never again recommend them to anyone..

14th November

The sites returned after I fired off that email yesterday (I assume they rebooted). My sites vanished again earlier today. And, about now 6:30pm, they have all gone again!!!

I'm taking legal advice on this!

Note sent to the suffering team in support:

AAAARGGGHHHH

That flipping server appears to be down AGAIN. I don't know appear you lot, but I am getting really, really ticked off about it. It's not and one site experiencing data loss, the lot are off the grid.

ERROR:> Can't read from control socket. Socket error = #10054.

No, really, I've had it. I've enough crepe paper in my life without this!

~ Paul

13th November: Webfusion are getting on my nerves!

Yet again, my sites hosted on Mango disappeared! Quote (from the Webfusion site):

Your site is precious.
Choose your host wisely"

Needless to say I emailed support...

Reply from support:

Hi,

Following the mango server rebuild it became apparent that some of the sites have not been restored correctly. However, some further file restoration work by the Leeds team appears to have resolved most of the problems.

I assume the reference to removing billing details indicates an intent to cancel your account - if this is the case you need to speak our Customer Services team, rather than Accounts, on 0115 917 0000 (select option 6), where they shall confirm your identity (for security reasons) and initiate the process to close down your billing details, transfer your domain name (where applicable) and remove you hosting account from our servers. The Customer Services team are available Monday to Friday, 9am - 6pm.

Once again, our sincere apologies for any inconvenience experienced.

Regards,

Adrian Jones
Pipex Support
PIPEX

Telephone: 0870 012 6600
Fax: 0115 877 0221

10th November: Continued patience my !

As you may know, if you've tried, most of my websites are down. Here's a few extracts from the support team website
[ Hosting Status for Pipex / Webfusion ]

I'm sorry, what patience? It was playing up several times last week - I reported it twice for a start - and it's been down all this week. If it's just a hardware fault / duff system then I find it reprehensible that an ISP of Pipex's size doesn't have a spare pre-configured server they can just plug right in and restore to. If the data is stuffed then the same applies, the cost of a RAID system as opposed to a single drive is miniscule, especially for a shared server.

Patient? Like heck, I'm ruddy fuming. These are just personal sites of mine, but if they were resold accounts (as they almost certainly will be for others) then I'd have scores of very, very angry people - justifiably so - wanting to know what the heck was going on…

Hi, yes, yes Mr Dixon, I know it's down. Losing £25,000 a day in sales, eh. That's not good. The engineers are working on it, soon have it back - any week now. Thank you for your continued patience.

Riiight!

If it was Ackadia down this long I'd go absolutely ballistic. You can be sure the insidiously long, carefully worded agreement reseller has a clause to the effect, Tough, we own you for the duration of the contract. Deal with it!
*Mutter*
Next year I'm moving all Ackadia and any other sites I have Stateside. Quarter the price, half the headaches, yep!



Update 12:40pm
The sites are finally back up, if a few days out of date due to the rollback, but the forum for ICU-Ouch is stuffed. … A few minute later and access if denied, again.

Woot! Everything is back and the forum has restored properly too. Now to get on with it!

9th November: Damnation!
Dadsense, ICU-Ouch and the rest are all offline after the (ISP's) server crashed and burned. For me, it is just irritating, for a busy online store it could be devastating. A little wake up call for the benefits of redundancy: It's not whether you can afford the extra cost of a backup server elsewhere, but whether you can afford not too, hmmm.

Either way, it's a gentle reminder to do your daily/weekly backups of online databases - blogs, forums etc. Yep, yep, yep!

9th November: Gremlin updates

For some reason Webfusion's servers are especially slow and glitchy, making it much harder than it need be. Webfusions' Mango fell over the other day, then Barium - hosting Ackadia went down - and again today (8th Nov), I think.
*Mutter*
Ackadia just died a death. Coming to something when I have to keep poking them to act. 99.9% uptime my .   Anyway…


*GROWL*
Either something is going on at my ISP, Webfusion, or something is flawed with the latest build of phpBB, because this forum keeps collapsing!

*Mutters, rips the lot out and start, yet again… *

Then, last night:

After a reboot we have encountered a hardware problem on one of the disks inside Mango. While this is not one of the main disks and as such will not result in the server or websites, email, FTP, etc being offline - it may cause some intermittent problems tonight and tomorrow morning whilst our engineers work on it. Apologies for any inconvenience.
… Update 11.53

Unfortunately the work the engineers were performing has not succeeded and the server will now require a full rebuild.

Ackadia is actually on another server - but half my other sites are here, and thus offline. So very angry!

Blasted gremlins...

Updates …
OK, I'm miffed, first the Forum went pear shaped, then something (or someone) reset the Photo Gallery to be unreadable (chmod 331 and 333). How annoying is that! Anyway, reset my passwords and ripped the lot out. Can't be too careful, nope!

Rebuilding notwithstanding, new sites I am working on include:

• DadSense
  Looking at making money with websites.
  
  
• [ www.icu-ouch.info ] ICU - Ouch!
  Help and support site for the victims of road traffic accidents, and related
  
  
• Smoking Related
  Some idiot annoyed me claiming, "Truth: passive smoking it good for your health". Filthy habit!