Visited 2502 times , 1 Visit today
Good news and bad news.
First off, the bad news.
The server our gallery was on experienced a fairly serious hard drive failure, along with issues rebuilding the raid array.
Fsck was not bringing back our data, and we had to go into full file recovery mode as we had not gotten to the point of making a backup on that server.
We were unable to recover image data to a workable degree.
Much of the image data we did recover, showed signs of the corruption.
Now that the bad news is out of the way.
We did manage to recover the integration code that allows our gallery to integrate with our user accounts and flockmod itself.
This allowed us to restart the gallery on a new server.
We now have backups of this code and that should help with allowing us to restart quicker should this sort of thing happen again.
We are looking into backup solutions for the images themselves, though given the speed of change and the size of the data it is slightly more difficult than the code.
In short, we had a near total hard drive failure within a week of launching the gallery.
This has caused us considerable downtime in attempting to recover the system.
We did not have backups of the system, as it was brand new.
We will try to provide more consistent ant redundancy in the future.
Rest assured that the actual drawing servers are a fair bit stronger then the gallery server.
Murphys law took full effect here though and we got hit in one of the few places we had not made strong yet.
We are extremely sorry about the downtime and confusion, hopefully the gallery’s new home will stave off these issues.
Comments are closed, but trackbacks and pingbacks are open.