This blog is intended to be a heads up.
TLDR:
- Hardware Upgrades "soon" but actual date unknown
- Database Upgrades even sooner, within the next 2 days
After the recent downtime where I had to do way too much data recovery, we put in plans for new infrastructure upgrades and improvements with help from a QnEZ Customer.
This customer invested a decent amount of money to help us upgrade our network capacity 10 to 40-fold, as well as adding some new servers.
When we finish figuring exactly how we are going to implement those services, we will need to impart downtime to install new networking hardware in each of the servers, as well as running new cables and installing a new switch.
Before we even get that far however we will need to take BigCloset and other sites offline, one by one, while we upgrade them from the backup MariaDB (MySQL) server they were using to our new MariaDB Galera Cluster which will bring improved reliability, speed, and recovery options.
In the past BigCloset had been making use of a redundant database cluster where one node accepted all the "writes" while the other two were read only, and while this worked fine in the past, it had been more than partially offline due to an admittedly preventable error, that I hadn't thought of and as such the new cluster instead of being "Master/Slave/Slave, will be a Master/Master/Master setup where any of the 3 (or more) servers in our cluster can be used to read or write data. This will allow us to much more efficiently make use of our distributed infrastructure while allowing us continued growth!
Comments
You are so awesome to us.
And thank you for the heads up.
BTW everybody, you might want to join the Discord chat channel so you can keep in touch with the rest of us while BCTS is down.
A Thank You
A big thank you to the QnEZ customer that enabled all of this to happen.
Thank you once again...
Robyn B
Sydney
Thank you
Not just for letting us all know in advance of the downtime, but for ALL that you all do to keep this corner of the Internet going. It means so much more than you'll ever know.
It's not much, but it's all I can afford at the moment... just a simple thank you. ---,--'-@
Hugs,
Roberta
A simple (to me) request?
After the last outage I have wondered why the site has not implemented an outage status option: A simple static home page which gives the reason for the outage, the current status and perhaps a projected new availability date and/or time, when known. Since it would have no other content and do no login or database activity it could be served from a single spare server.
My own site does some of this. A single switch behind the home page provides a "maintenance" banner and turns off login. Of course there are alternative ways and means for admins to actually access the site :)
Penny
Using DNS, you can use a server somewhere else totally.
DNS allows for a backup and when the site is down, you can have a secondary location with the static page listed. It can be offsite and hosted as a deep page on some other website that no one knows about. http:\\Sitename.com\our_hidden_pagename.
I would still recommend that
I would still recommend that there be a slave server (or even secondary service on a box that's not normally used for SQL). That's your live backup, that can be copied off-site without impacting the master system or cluster.
+1 on you for being proactive, of course.
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
N-1 All Master Galera
While all servers will be masters, we will likely maintain an N+1 setup where that extra server will always be live, and always be read/write but can also be instant failover if there is a problem.
Orwellian
To paraphrase:
"All Masters are equal but some are more equal than others."
VPN
Are you still offering the VPN service, and how would these downtimes affect it. Been thinking about it, contingent on getting my SS straightened out.
“When a clown moves into a palace, he doesn’t become a king. The palace becomes a circus.” - Turkish Proverb