Kassblog

We got all our users up and running on the new, single-domain servers this morning. For a disaster recovery situation, 24 hours was not too bad. How many tries did this take to get right?

Days after recovering from our failed network migration, we have had another server blowup. The air conditioning system in our server room failed this weekend, the room heated to 150 degrees F, and our Exchange server drives failed. We did get a good backup of the data store last night before the failure, but now we are left without both an Exchange server and the internal expertise to reinstall Exchange in a multi-domain environment.

At the same time, I have gathered a lot of feedback from other BAISNet tech directors and discovered that we are the only school among the respondents that is running multiple Active Directory domains. So, in this state of emergency, we have decided to immediately rebuild our new servers in a single domain environment, test it this afternoon, and migrate users to it tomorrow.

The simplified approach has reaped dividends within the first hour of adoption. Our systems administrator already worked up the primary domain controller the other day, so all he has to do now is to prep the domain for Exchange and install that. Without the multiple domain issues, we should be able to test the new server environment with user accounts by this afternoon. Our tech department users will spend the night logged into the new servers, and we will migrate others users if all is still well tomorrow morning. We may even recover all mail if we can mount the backed-up data store in an offline Exchange server and spin off the PSTs one final time.

Wish us good luck.

This feels familiar. We have twice this week unsuccessfully attempted to migrate to new servers. The problems all involve Exchange server communication between parent and child domains. I finally decided to pull in some external expertise and discovered that we are doing things the hard way at a couple of levels. First, our four-domain architecture runs counter to the conventional wisdom for organizations of our size and complexity. Microsoft and our consultant recommend a one domain architecture with organizational unit/group policy management of user privileges. Second, we manually complete a number of steps that the consultant has figured out ways to automate or shortcut. For example, we spend 20 minutes per user unjoining them from the old domain, joining them to the new domain, and then setting up their new profile. Apparently, there is a way to edit the user’s registry to move a user to a new domain without changing their profile. After the third attempt to reinstall and correctly configure our new servers, we rolled back to the old servers and returned to the drawing board.

On another note, I confirmed that my struggles obtaining PERL debugging information in the browser is a new IIS6 security feature. I found a description of the problem that indicates that there is not a good way around this feature. Too bad that I have not yet found error handling options in Activestate PERL similar to those found in PHP. While it was convenient to see error messages when PERL scripts did not compile correctly, this forces me to get a legitimate PERL development enrivonment that can provide debugging information live and write my scripts to capture errors on the fly. These are good things.

Today, I continued work on our new WIMP server configuration. As opposed to LAMP, or even WAMP, WIMP stands for:

Windows
IIS
MySQL
PHP/PERL

Our WIMPy setup is coming along just fine, aside from a few speedbumps. The first occurred when PHP repeatedly failed to load the php_mysql.dll library. It took me three hours to realize that PHP was pointing to \Windows instead of \php to find the php.ini configuration file. This despite the fact that php.ini was only located in \php, and I had added that to the PATH server environment variable! A test script with the phpinfo() command was essential to discover this fault.

I am still working on the second roadblock. PERL is not returning informative error messages. When a script does not compile properly, the browser only returns an “incomplete set of headers” error message. In our old IIS5 setup, this would usually be followed by “the headers it did return are …”, which would provide the substance of the error message. Not this time! I need to determine whether this is a normal consequence of an incomplete set of PERL modules, or whether there is a way to turn on more verbose error messaging. I suspect that my answer lies in some command-line testing.

In other news, Richard Bender has completed most of the new server setup and will begin the migration tomorrow. Ina upgraded Raiser’s Edge to version seven, and we are aiming to join the first admin users to the new domains on Monday.

Our beloved student.user network account graduated today. It first entered the school in the fall of 2002 as a test account for student network privileges. Needing a security group, it joined the new class of 2006. For four years, student.user made its way through the ninth grade curriculum, found a new group of friends during sophomore year, unsuccessfully ran for V.P. of Diversity junior year, and experienced a vicious senior slump this year. College really isn’t in the cards for student.user. We think it is going to take some well-deserved time off and contemplate what to do next. A return to the class of 2010 is pretty much out of the question, since our current practice is to create a student test user account for each class. So, 2008.user and 2009.user are currently making their way through the rigorous UHS program, and the tech department will have many graduations to celebrate in the future. One thing for sure — the tuition payments are killing us!

p.s. Teacher.User appears to be a lifer.

Today, the Class of 2006 graduated from San Francisco University High School. On Monday, I will continue our four-year tradition of disabling the students’ network accounts two days after graduation. This is one of the more unpopular policies I have. Many students would like me to keep the accounts active through the summer, until they have established their college accounts.

Fellow tech directors, what policies do you follow for graduating seniors?

Arguments for closing accounts right away

You don’t attend the school anymore.

You’ve graduated. It’s time to let go and move on.

Acceptable use agreements no longer govern account use.

We need the summer to remove old accounts, reclaim disk space, and establish new accounts.

Email services cannot exist independently of other network services.

Some colleges have already made network accounts available to new students.

Most of you already have personal email accounts.

Take this opportunity to update your UHS Alumni profile with a permanent email address!

Arguments for keeping network accounts active through August

Not all colleges make network accounts available right away.

You deserve the opportunity to reach each other easily before you go to college.

Departing seniors can be trusted to use accounts appropriately for the duration of the summer.

It doesn’t take that much work to perform the necessary account management.

You might use your network accounts to participate in forum discussions.

It’s just plain mean to suspend the accounts right away.

I am starting to build our our new web server today. It is exciting to start from a completely blank slate!

Over the winter break, I built out this web server once, when we thought we were going to launch the new servers then. Since we postponed to summer, I have the opportunity to learn from that experience and create a cleaner install this time around. Last time, I thought I would install PERL, PHP, and mySQL on the C: (system) drive in order to keep all applications there. This time, I am going to reserve C: for the operating system and default Windows applications and put everything web-related on D:. This way, we can image D: once before launch and keep that image as a snapshot of the web server configuration in case of disaster recovery. Also, if the system ever has problems, we can wipe and re-image the C: drive without affecting the web server contents.

Blue = blocked (rate control)
Red = blocked (spam)
Green = allowed

Our filter tells us that most of our incoming mail is spam. Though I am happy to have a spam appliance (from Barracuda), the overall increase in spam volume leads to a corresponding increase in false negatives. Users start receiving several spam messages each day, reminiscent of a few years ago when that is all the spam that existed. It is a shame that we have not been able to pass effective anti-spam legislation to thwart this problem.

While in Botswana last week, I was introduced to a class of history students as the person who first introduced student email accounts to Maru-a-Pula School. To my surprise, the students applauded! At University High School, several students have taken the time to thank me for their accounts. Yes, it requires more work (though not as much as you might think) to manage 400 (or more) additional email accounts. Students do tend to come to school with a personal account at another service. However, school email accounts provide added value in several ways.

A user-friendly email system for those who have an unfriendly one

A standardized naming convention so that students are easy to find

A searchable email directory, such as Exchange Global Address Book

An easy way to create class mailing lists

The foundation for a culture of open, rapid communication among teachers, staff, students, and parents.

A “professional” address for students to use with the outside world, such as colleges and prospective employers

An easy way to transfer files between school and home

A straightforward link into email-enabled community web software such as Moodle and bulletin boards

… and perhaps most importantly,

A sense of pride that the school trusts them with a branded email address

The last item has convinced me the most of the value of a student email system. The enthusiasm and pride is evident when the school helps students communicate more esaily and effectively. The gratitude is palpable.

To make administration easier, choose an email system that integrates with your authentication system. Set up mailbox size limits to avoid filling up your server or overloading your backup system. Allow a variety of protocols for reading mail, such as webmail, POP, IMAP, and/or RPC over HTTP (if your sysadmin feels they are secure enough). Document procedures for students to solve common problems, such as how to fix a full mailbox or set up an IMAP client.

We have learned one more lesson about the server migration: it ain’t over ’till it’s over! In the stretch run of the installation, we ran into a number of unanticipated time sinks with regard to setting up groups, folder permissions, and DNS. As a result, we decided it would be better to more thoroughly test the migration during the spring semester and cut over in June than to sick 500 users onto immature servers this week. As there was no critical reason to move to the new servers now, the main consequence of this change is that we have had to “unprepare” our users for the switch. Users who had anticipated downtime and new passwords are now surprised to find that the status quo remains in place.