Site Outages... Planned and Unplanned

Hi everyone,

I’m sure you’ve noticed site performance has been a bit patchy over the past few days. There’s a number of contributing factors to this (all relating to our server’s hard disk system) and it’s a bit of a circular story so I hope this makes sense

We had been noticing that the site has been getting slower a little bit at a time, slow enough sometimes now that during peak times we were unable to serve all requests. We made a bunch of small changes with the help of our hosting company but realised we need a big upgrade to our disk clustering software – which we are planning to do Monday the 27th at around 1pm AEST with a roughly 2 hour downtime.

Ahead of this upgrade however we need to create a full mirror of our existing file system. We hoped we could do that in the background while still keeping the site up, but that gave us the big chunk of downtime earlier today so we had to cancel that attempt.

Our hosting company is now trying to figure out another way to create this mirror, but we may need to take the site back down again. When we get the estimates on how long it will take and when we can do it, I’ll update this thread.

• we’re 95% sure we’ll be taking the site down for 2 hours on monday
• we’re pretty sure we’ll have to take it down a second (or first?) time to prepare for the upgrade and we’ll update you on that as soon as we know more
• there may be some more windows of just general site slowdowns/robot pages until we’ve done this upgrades as we’re pushing our disks too hard

Thanks very much for being patient with us on this. Annoyingly, it’s keeping me from working on redesign stuff (but don’t worry Stu, James, and Erin are making really good progress – i just don’t want to miss out on the fun)

Hey John,

These things happen and I am glad you guys are so open and honest with us. I know it must be a lot of work at the moment, and I am sure most of us understand. Keep it up devs

I wonder if I’ll ever be able to actually manage a website like you guys do. Thanks for the info!

Way to keep everything in control. Good job devs!

Thanks for the transparency John

Yes agreed, thanks for the updates. I’m sure there is plenty of opportunity in the future to learn more about clustering and caching… with a network this large now.

Remember that you need rest, sleep…

Thanks for the update and good luck.

I don’t envy massive website scaling issues like you guys have, but I trust you guys/gals are all working hard to make the best decisions. Keep up the good work and thanks for the update

Yeah, I had been noticing the slowness over the past few days. Thanks for keeping us informed. Good luck!