JonB2 Forum Administrator Offline I Love YaBB! Posts: 15 Joined: Oct 16th, 2014 Mood: Aggravated Yabbforum.com is down Mar 2nd, 2015 at 5:44pm on scheduled reboot, I got a Linux version of the BSOD - could not mount filesystem I do have good backups, and the drive is probably NOT toast although I cannot mount it right now. Bad day at the OK corral Plus I have a whacko on my case about her broken wi-fi Arrrrgh! IP Logged
Bill Myers Tester Offline Using YaBB since 2002 Posts: 89 Location: Los Angeles, CA Joined: Feb 13th, 2014 Gender: Awards: 1 Mood: Blessed Zodiac sign: Re: Yabbforum.com is down Reply #1 - Mar 2nd, 2015 at 5:56pm Thanks for the update. If anybody can fix this, you can! Morning, noon, or night, have a great one! WWW Facebook Twitter IP Logged
JonB2 Forum Administrator Offline I Love YaBB! Posts: 15 Joined: Oct 16th, 2014 Mood: Aggravated Re: Yabbforum.com is down Reply #2 - Mar 2nd, 2015 at 6:51pm OK, I have found what is wrong and it is foogly, The LVM space (that's where the data is stored) is forked. Somehow the Volume is now tagged as Inactive. I need to transfer the contents off before I try to repair Arrrrrrrrrrrrrggggh! IP Logged
JonB2 Forum Administrator Offline I Love YaBB! Posts: 15 Joined: Oct 16th, 2014 Mood: Aggravated Re: Yabbforum.com is down Reply #3 - Mar 2nd, 2015 at 10:48pm Very very very ugly - non-fixable, but recoverable. Which translates to I am tarring off containers to USB drive using a Linux rescue CD. That means I should have all the data, even since the last backups. BUT the OS has to be restored. I found that the problem is a known kernel bug in Fedora. YAY! I am so happy it is 'known' but unresolved still 2-3 years since it appeared. It also means, I have no idea how it happened... IP Logged
Dandello Forum AdministratorYaBB Modder Offline I love YaBB 2.7! Posts: 2234 Location: The Land of YaBB Joined: Feb 12th, 2014 Gender: Mood: Annoyed Zodiac sign: Re: Yabbforum.com is down Reply #4 - Mar 2nd, 2015 at 11:08pm Oddly enough, we've had similar issues with a USB harddrive in Win7 Pro after an iTunes update. No idea how it changed the drive to inactive. At least in our case it wasn't the internal boot drive. Perfection is not possible. Excellence, however, is excellent. WWW IP Logged
Dandello Forum AdministratorYaBB Modder Offline I love YaBB 2.7! Posts: 2234 Location: The Land of YaBB Joined: Feb 12th, 2014 Gender: Mood: Annoyed Zodiac sign: Re: Yabbforum.com is down Reply #5 - Mar 3rd, 2015 at 12:05am Just talked to JonB - This is a major OS/hardware issue that is going to take some time to work through. On top of this, JonB has real live work to handle. So please be patient. Perfection is not possible. Excellence, however, is excellent. WWW IP Logged
Bill Myers Tester Offline Using YaBB since 2002 Posts: 89 Location: Los Angeles, CA Joined: Feb 13th, 2014 Gender: Awards: 1 Mood: Blessed Zodiac sign: Re: Yabbforum.com is down Reply #6 - Mar 3rd, 2015 at 12:20am Thanks for updating us about it. I hate when this kind of stuff happens; to me, or to anyone else. On a note about downtime that I hope you'll consider as a future reference kind of thing, perhaps the Admin Center for 2.6.11 and future updates can already have links in the admin help section that includes this forum of yours as a backup forum in the event of extended downtime that YaBB's forum may experience. I've already done this in my own forum, and those quick links are nice to have available. Think of it in the way that Jon offers up support links on YaBB's main page that includes your forum here. Yes, I know that his main page is down right now because of what's going on, but you get my point ... just adding a bit of redundancy if you will. Morning, noon, or night, have a great one! WWW Facebook Twitter IP Logged
Dandello Forum AdministratorYaBB Modder Offline I love YaBB 2.7! Posts: 2234 Location: The Land of YaBB Joined: Feb 12th, 2014 Gender: Mood: Annoyed Zodiac sign: Re: Yabbforum.com is down Reply #7 - Mar 3rd, 2015 at 12:30am We have been working on the code to do what you've suggested - but we need to run it past Corey before implementing it. Perfection is not possible. Excellence, however, is excellent. WWW IP Logged
JonB2 Forum Administrator Offline I Love YaBB! Posts: 15 Joined: Oct 16th, 2014 Mood: Aggravated Re: Yabbforum.com is down Reply #8 - Mar 3rd, 2015 at 3:31am I was able to restore the server to new drives from our last disk image backup. There's a fair amount of other data (say 50 GB) that has to be shoehorned back on from the backups I made ealier. AMAZINGLY enough, ALL the backups ran to the USB drive last night! AND I got Boardmod & YaBBForum with tarballs as they were when the system was shutdown. So we are in OK shape IP Logged
XTC Global Moderator Offline YaBB 2.6.0/1/11 Mods Posts: 174 Location: @ustria Joined: Feb 12th, 2014 Gender: Mood: Freaky Zodiac sign: Re: Yabbforum.com is down Reply #9 - Mar 3rd, 2015 at 5:15am JonB2 wrote on Mar 3rd, 2015 at 3:31am:So we are in OK shape Well done—you've worked very hard! lg XTC IP Logged
JonB2 Forum Administrator Offline I Love YaBB! Posts: 15 Joined: Oct 16th, 2014 Mood: Aggravated Re: Yabbforum.com is down Reply #10 - Mar 3rd, 2015 at 10:46am OK, the restorations went waaay faster than I expected. We seems to be back up and in good order. Please check things out and test. Once things are 'for-sure' right, I will get a new server image Thanks very much, this morning made up for yesterday, which was the Monday from Hell Edited: I did learn something, for sure, when you do a shutdown/restart - it would be a blessing to have set Maintenance Mode first - otherwise , if you have to restore, the store doors are open IP Logged