The Forums Have Broken!

The most amazing thing happened to our most successful online service. It broke. Our online forums became corrupted when a web server log file “filled up.” Probably two-thirds of all messages disappeared, and people were unable to post for a couple of days. Coincidentally, our backup system experienced global catalog corruption that lasted several days, making it impossible to restore the forums from backup. It seems so much like divine intervention …

The kids have pretty much taken it in stride. They get more out of present conversation than stored messages. So perhaps this will not have much impact on the activity and character of the forums. Or maybe it will give the students the opportunity to rebuild the forums in a different form than the original.

Lessons learned:

1. Pay more attention to our backup system. Even when it indicates that it is working properly, it may not be. Test restores are the only way to be sure that the backup system is running well.
2. Why should a log file “fill up” without warning? This seems at least like a fundamental flaw in IIS and definitely another reason to move to Apache on Linux in January.
3. Flat file databases are more vulnerable to corruption than SQL. We run dozens of web services, but the forums were the only one corrupted by this failure. Of course, our next forum software will run on mySQL.

Comments are closed.