Update - Apr 7 4:51 PM PDT - The site is back, in most of it's glory. The problem turned out to be the voting module not being compatible with the latest release of the database software. So, no voting on stories until I figure out a replacement module or they update their code. And no reporting on old votes, either. Sorry.
Most everything else has been turned back on (or will be) except: reporting comments to moderator (flag for administrator) - that module conflicted with the one that makes Random 5olos possible so it had to be taken up into the mountains and sacrificed to an angry god, sorry. Also, the little clock at the top of the right column that showed what time the server thought it was had a problem when the updates to the search index was run, don't ask me how but the clock interfered and until someone writes better code than my own jackleg module (I wrote that one) we'll have to do without the clock. Isaac says hi.
Many thanks to Bob who did the major work on this debug. We should buy him a bag of chocolate or something, he likes chocolate. :)
Hugs,
- Erin
Update - Apr 7 5:55 PM EDT - I'm getting more confident that we've finally located the problem or at least the main one anyway. Sorry for the short outages here and there as I've tweaked the server settings to handle the extra load the steady growth of the site has placed on the server. More mysql connections and more web server daemon resources are making things run smoother than they have before. Erin still has to turn some of the other modules and site features back on but I'm happy at the moment that the site isn't crashing like it was before. - Bob
Apr 7 4:15 PM EDT - I think I finally got a lead on the problem thanks to some commands in a utility to monitor the mysql server. I had to turn off the voting system since it looks like that process was stuck on when the system crashed. Not certain if it was just a symptom or the actual cause though. - Bob
We're back on the main server since it does not seem to be a memory problem after all. We had the same problems on a completely different server, which means it has to be a software problem. Bob and I are going to continue working on it so expect intermittent outages.
Hugs,
Erin
Comments
Take your time with it
so it gets fixed right. I think I speak for everyone who has anything to do with Big Closet Top Shelf when I say, THANK YOU, both to you Erin, and also to Bob. All your hard work and dedication to keeping your sites up and running makes it possible for those of us who either can't, or are afraid to, get out and meet and interact. It provides a much needed forum for those of us who fancy ourselves to be writers, and enables us to communicate on several levels, with people who think the way we do...who have the same points of view...who understand, support, commiserate, and comfort one another.
THANK YOU BOTH, AND ALL THE VOLUNTEERS WHO WORK SO HARD.
Hugs and love,
Catherine Linda Michel
As a T-woman, I do have a Y chromosome... it's just in cursive, pink script.
I hope it works out fine!
The more recent changes were some of my favourite. Both QuickCuts and Random 5olos have proven to be popular, and it'll be sad if they have to go.
On a side note there is a couple broken links at the top of the home page - namely the "CrystalHall" - for Whateley, I presume - and the "Tor der Traeume" - whatever this one means. Just thought I should give you a heads-up.
Faraway
Big Closet Top Shelf
Where you can fool around like you want to and most you get is some bemused good ribbing!
Faraway
Big Closet Top Shelf
Where you can fool around like you want to and most you get is some bemused good ribbing!
What's going on
What we're doing is turning things off, one by one, to see what makes a difference to the site functioning. So far, we've made small improvements and found problems we didn't even know we had but nothing that solves the big hanging the server problem.
So, we keep doing things. Don't be surprised as things disappear one by one while we work.
Unfortunately, both Bob and I have to be gone for a period this afternoon, so if the server hangs between 130 pm and 4 pm Pacific time, there may be no one online to give it a boot. I'll have access to a computer but I may not be able to get time to look at the site then.
So, be patient.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
This reminds me of the simulator scene in Apollo 13
...where Gary Sinese is on his back, turning all the switches on and off to see how to get enough power without blowing the whole system... Time consuming to say the least. Thank you once again for all of your efforts to make this the best site on the net!
She was born for all the wrong reasons but grew up for all the right ones.
Con grande amore e di affetto, Andrea Lena
Love, Andrea Lena
Right now, I can't commit
Right now, I can't commit days/hours/weeks of continuous effort to the cause, as you do, but I can spend a few hours over the next day or so, if you want to give me a copy of the base configuration files and site. I don't need the database for that :)
One thing I _did_ notice was that one of the external load links bogged down right as the site did, this last time.
You can also edit a cron job to kick the apache server every 15 minutes or so. If you use apache2ctl or apachectl restart, it shouldn't even be noticed by the users. (I did this as an emergency workaround)
BW
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
MySQL
Doing a restart of Apache without doing a restart of MySQL does not solve the problem. Right now, we do a restart of MySQL and Apache via cron job every thirty minutes, this is because of a known memory leak which may well be the culprit. We're trying to pin that down.
I'm not sure what you mean by "copy of base config files and site". At any rate, getting the time to package something like that up to send someone isn't going to happen today. Sorry.
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Okay - if restarting apache
Okay - if restarting apache without restarting mysql doesn't fix the problem, then it's definitely a glitch in the way that mysql is responding. Memory leaks are also a problem. You might want to try running mysqltuner, just to see if it throws any additional information up. http://blog.mysqltuner.com/
I wouldn't expect it to be done _now_, or even today. You have something to do this afternoon, obviously.
Configuration files are the php.ini file used by apache, the mysql.cnf (or whatever your distro calls it) file, and the core drupal files. (if possible). I doubt the apache config files would be helpful.
Since the database itself hasn't changed, I certainly don't need that. I'd just be going through looking at where things might be bogging down, and plugin interactions. I haven't done programming in years, but I can debug, and can rework other people's code reasonably well. (which is weird, when you get down to it)
Some other things to check.
'netstat -an | grep :80 | wc -l '
That'll show you how many apache connections are open
'ps aufwx | grep apache (or httpd, whichever shows up) | wc -l ' tells you how many apache children are going. On my system, it turned out that I needed more apache children, because I was running at 900+ connections at all times. Once I went to 300 children, the server stabilized. (I was having lockup issues almost identical to these)
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
"Once I went to 300 children"
"Once I went to 300 children"
I wonder if that's where Cathy is headed? And she doesn't have any Apache children yet; that'd be another challenge. :-)
Kris
Kris
{I leave a trail of Kudos as I browse the site. Be careful where you step!}
She might go for Picts.
She might go for Picts. Currently, she's just collecting Angles and Saxons. Maybe a couple of them are Jutes. How about a Fomorian?
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
"intermittent outages"
Story of my life.
Angela Rasch (Jill M I)
Angela Rasch (Jill M I)
Thank you
Thank you both for the updates and for working hard on the site. We all appreciate all that the two of you are doing!
-Tiffany :-)
Much Appreciated
What more evidence do we all need of the time and effort that Bob and Erin put into this site?
We do appreciate it, don't we?
Surely we can find it within our collective hearts to make it so that the site is not subsidised by the dedicated people who run it. Let's contribute a little. I know some of us can't, but some of us can.
Come on out there,
Joanne
Thanks to you both!
Thanks a lot Erin and Bob! You did it! BC is back yay!
hugs!
Grover
Server problems.
I believe in you, Erin. Bob too.
Right up till we see the screen that says 'All your stories are bleong to us!"
:0
You'll get this one, just like you have all the others.
Sarah Lynn
re: back up on site
im having trouble returning to original page after reading a story, i have to refresh page each time. is this something you can cure, if not i can i can live with it.
robert
Not sure what you mean
I can't quite figure out what you mean by what you say. Can you PM me with a more exact description?
Hugs,
Erin
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.
Well, that explains the
Well, that explains the behaviour of the voting recently. The last time I voted, it actually _deducted_ a vote, rather than adding two (which is what my voting was doing 'normally')
Glad you tracked it down.
BW
I'll get a life when it's proven and substantiated to be better than what I'm currently experiencing.
Thanks
Erin and Bob
What would we do without you? Thank you for all the time and effort you've put into getting things running again, and for keeping us updated on your progress.
Hopefully it won't be too long before all the hair you've undoubtedly pulled over the past few days grows back. As my hairdresser says, the difference between a good haircut and a bad haircut is about a week.
Positive Support
Bike Archive
Bike Resources
Quicklinks gone?
I notice that the QuickLinks, etc. (the stuff that was under Edeyn's name) are gone from the top of the home page (using the Reversed Theme). Is that part of what was causing the problems and will it be back? I don't use them and so won't miss them, but I am curious.
Karen J.
"Being a girl is wonderful and to torture someone into that would be like the exact opposite of what it's like. I don’t know how anyone could act that way." College Girl - poetheather
“When a clown moves into a palace, he doesn’t become a king. The palace becomes a circus.” - Turkish Proverb
Testing
As Erin said a few comments above, she and Bob are still testing things by temporarily removing a couple of features, waiting to see if it makes any difference, and if it doesn't, putting them back.
They've already determined the voting module doesn't play nicely with the latest database update, so until it's updated, it's disabled. Similarly, the "Report to admin" link and the little green clock letting us know what the server time is weren't playing nicely, so have also gone.
Presumably once the site is stabilised, they can actually spend some time investigating possible updates / replacements for the voting module, "Report to admin" link and server time clock. I just had a quick peek over at Drupal's site, and it appears as though there are numerous different voting modules that hook into the Voting API, each implementing different features, and each developed on different versions of Drupal (ranging from 4.x to 6.x, with some dev builds for 7.x).
Unfortunately, the extensibility of Drupal is one of it's greatest strengths...and weaknesses. it is almost infinitely customisable and extensible, with hundreds (thousands?) of different modules written by different developers - so inevitably there are occasions when some modules don't play nicely with others or the underlying database. Obviously Erin and Bob's first priority is to get the site running as smoothly as possible with as few modules disabled as possible - only then can they take the time to examine the misbehaving modules and search for updates / submit bug reports etc.
As the right side of the brain controls the left side of the body, then only left-handers are in their right mind!