Corrupted table

Printer-friendly version

Author: 

Taxonomy upgrade extras: 

If you couldn't log on earlier this morning, there was a corrupted users table, due to an earlier crash. It's fixed now. :)

Hugs,
Erin

Comments

Scary...

Andrea Lena's picture

...it was like walking into the A&P and finding all the shelves and displays still standing but empty.

  

To be alive is to be vulnerable. Madeleine L'Engle
Love, Andrea Lena

Well, I'm glad I got up early today

erin's picture

I got up a half hour early today, so I found it quickly. Seems to have happened between 2 and 6 am my time. Some sort of soft crash didn't quite take the server down but caused a miswrite to the user table which corrupted it. We're investigating to see if anything is wrong with the drive or drive controller.

Hugs,
Erin

= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.

Phew!

When I couldn't log in and the site was claiming that it couldn't recognise either my username or my email address, I was desperately hoping it was a site problem and I haven't inadvertently annoyed someone important :)


As the right side of the brain controls the left side of the body, then only left-handers are in their right mind!

I noticed at the time that

I noticed at the time that the number of online users dwindled down to zero. I have never really payed attention to it before so I was hoping that it indicated there was some kind of problem. It didn't seem likely that no one at all would be on with a site this large.

logon

PattieBFine's picture

the funny thing was... it didn't JUST not let me login... I couldn't READ anything on the site as well. Attempting to open a link and read, told me I wasn't authorized. Said I had to be loged in to even View a story. Wacked login never kept me from just viewing before. Good lords!... I nearly was in a PANIC!

Huggles!

Thanks for all your hard work...

NOW GET SOME REST!

OR ELSE!

:-*

Much obliged

Thanks for fixing it so promptly.

When it wouldn't let me log in, I was worried that I'd forgotten my password, and I can't do one of those things where you get emailed a password fixing thing because my email is down and my internet provider is not as fast at fixing things as you - the system failed on Friday, and they still haven't fixed it.

I was wondering...

Wendy Jean's picture

If I had done or said something. Glad it is fixed, now get thee well.

An Admin's job is never done & a muse on my late uncle

Good work, Erin.

People forget how much work goes on behind the scenes to keep it all running. PNATTTMBTC sure applies to the Admin duties. IE Pay no attention to that man behind the curtain.

http://www.youtube.com/watch?v=YWyCCJ6B2WE

Just wondering, how is/did the server tests go? Is your ailing machine on the mend or headed to the great recycling bin in the sky?

John in Wauwatosa

P.S. Sorry about the dark humor but was at an uncle's funeral on Saturday. He died December 10th. He would have turned 90 on December 19th, the day AFTER I will turn 55. I will miss him. I was very lucky to have four fine uncles and four wonderful aunts. Just my dad's older brother and their older sister left. My aunt and uncle were married over 67 years at the time of his death Monday.

John in Wauwatosa

The server that we were

Piper's picture

The server that we were having issues with, is currently DEAD.

The PSU went, and took out two HDD's and possibly the controller card.

We are currently working on getting replacements in, but for the time being, sites are being hosted on other servers temporarily.

-P/KAF/PT


"She was like a butterfly, full of color and vibrancy when she chose to open her wings, yet hardly visible when she closed them."
— Geraldine Brooks


RAID controller? What type?

RAID controller? What type?


I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.

Redundancy

Cripes! You occasionally hear of voltage spikes between the mains and computer, hence one of the lesser-known functions of UPS units is to smooth out the line voltage; but it's very rare you hear of the PSU creating a spike, which of course has the potential to damage any / all of the components. Given everything's back up, presumably you either had backups which could be implemented elsewhere quickly, possibly combined with a RAID 6 array on the main server (given that RAID 6 is the minimum RAID level that can cope with two dead drives)...

Unfortuantely, as a sysadmin you can never be too careful about planning for disaster recovery - skimping on redundancy / backups / RAID is generally not a good idea :D


As the right side of the brain controls the left side of the body, then only left-handers are in their right mind!

Not that uncommon

Bestec power supplies, in a notorious run provided to E-Machines and HP, would blow, killing the motherboard - but not the hard drive, processor, or any other peripherals.

The breaker function internally is supposed to try to stop that from happening, but almost all manufacturers of power supplies cut massive corners to save pennies. Antec, for example, does fantastic designs. Then they pull caps out of the units until they stop working, then add one back in. They then use Fuhjyyu and other craptastic capacitors to build the power supplies. Rubycon, Samsung, even United Chemicon are VERY rare to see in consumer power supplies. Instead, it's Teapo (so-so), Capxxon, and the aforementioned Fuhjyyu. These are 'two years, at best' capacitors.

There _are_ some brands that don't suffer from it. PC Power and Cooling tends to overbuild their power supplies - it's what I use in machines I don't want to die.

Now, what's shocking me is that the surge sounds like it was on the 12 volt rail, rather than the 5 volt rail. Usually, unless you see lightning coming out of the supply, the peripherals tend to survive. (not necessarily the raid controller, however)

(Yes, I HAVE seen lightning come out of a power supply. That was fun, until I could get it unplugged. You could watch the light show from outside and have a good idea of what part of the board the arcing was following)


I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.

Data Recovery and Backups

Piper's picture

On that server, we were backing up to the server every every 2 hours, plus a nightly backup that went off-server (but not off site).. The problem came in to play that too-much roll-back would have happened between the last off-server backup, so I had to have the data-center disable SMART on one of the other servers, install the drive in an empty bay, while Is spent a lot of time working to get the drive mountable (read only) to remove the most up2date backup.

Our current plan is to build a MySQL Cluster for most our servers that will allow us to have built-in instant redundancy for all our sites.

-P


"She was like a butterfly, full of color and vibrancy when she chose to open her wings, yet hardly visible when she closed them."
— Geraldine Brooks


No panic here.

Extravagance's picture

I just kept calm and carried on. = )

Precisely what I carried on doing, I shall leave to your imagination. ;)

Catfolk Pride.PNG

Sorry!!!

Andrea Lena's picture
0515_sad-waitress_400x400.jpg
You'll hafta wait.
I'm on my break...and besides;
that's not my table!!!

  

To be alive is to be vulnerable. Madeleine L'Engle
Love, Andrea Lena

Groan!

Sorry is right, go to the corner, bad punster.