So I rebuilt Alvis on an office machine today. I ended up copying Kulshan-new directly (using this cool thing called UDPcast). I spent a while this evening configuring Apache2, MySQL, CVS, and CAS. It’s kind of a skeleton right now but you can use CVS (it’s a copy of Alvis’ repository), webupdate, scriptupdate, and Apache seems to be okay. It uses itself as a database. It is using a copy of Odin-new’s database (which is a slightly-modified copy of Kulshan’s).

Once we install the new servers on Thursday, I can temporarily take Alvis out of production and swap disks or do a level0 dump onto it or something of that sort. Of course, that is more complicated than it should be because Alvis is serving student DNS and a MySQL for Departments. But it shouldn’t be too much work to work around that.

That being said, some transitions are due soon. Student DNS and Proxy might be the first two.

Few days ago I set up IMAP, POP3, and SMTP on the box that will replace Kulshan. POP3 and IMAP are secure. Of course I was able to log in and such, and receive/send mail and all. But it needs some tweaking because Kulshan ran Dovecot but the new box is running Courier. So the Maildirs need to be tweaked a little bit and I’ll have to look into that so people can keep all their millions of saved messages when we transition.

EDIT: I decided to go with Dovecot instead. Copying your Maildir over works fine now.

Today I set up the dedicated MySQL server. Kulshan’s databases have been dumped onto Odin-new. I modified the privileges tables. They currently need to be secured. I organized them in such a way that each user has only one password no matter what host it is coming from. This is not how we did it before. Anyways, it’s a lot cleaner. While all hosts connect using the same password, tomorrow I’m going to limit where users can connect from so certain users are restricted to coming from certain hosts.

All of the hosts will have the same database password lists, except I will modify each one so that it only contains the passwords that particular host needs.

Right now everything looks pretty nice. And also phpMyAdmin has a new look.

I tested out my modified version of makeDhcpConfig on server1, and it wrote a proper file, connecting to Odin-new’s database. Things are coming together.

After all of the servers are complete, I’m going to move more toward security and get them tightened up. I’ve been keeping that in mind, so they’re pretty clean so far, but some firewalls and such would not hurt. FreeBSD does a nice job of configuring things securely by default, too.

This morning I woke up to errors indicating that all of our servers had been down. I logged in and started some services which aren’t started automatically on boot and figured it was a hiccup in Bond Hall. The outage occurred a little bit later too and Nick called Rick. Apparently there was some confusion over what power to cut. Rick got things worked out and services seem to be okay.

I started scripting something for Vidar and Vali to run via cron to exchange temporary DHCP leases. I got mostly done but I decided to do some research on DHCP failover. It turns out that in version 3 (3.0.3 is current) of ISC’s DHCP server, they introduced Failover capability. So it seems that I might have to do some more research (the online documentation is quite sparse) and get that set up.

If DHCP can be setup to failover, then that simplifies things. Vidar and Vali will already both be running DNS. LDAP has a built-in client-side feature to try another host if the first one times out. Squid can be set up too. If I end up doing this, that would eliminate the need for CARP entirely. CARP isn’t very effective anyway because individual services won’t failover, only an entire interface if the peer is not responding. And also, transferring dhcpd.leases from master to backup using scp is kind of an ugly hack, so I hope this service-level redundancy proves possible after more research.

I also want this hat:

While LDAP and SLAPD were configured, things needed some work (to get TLS working). So I did that tonight. I haven’t dealt much in that area so it was a lot of experimentation. It seems to be working on Server1 now using a self-signed certificate.

The clients needed to be configured (which was confusing because there are more than one ldap.conf files). Anyway I configured my own computer to use ldaps://server1 and sniffed using Ethereal. After analyzing the TLS authentication process, I noticed it was an error relating to the Certificate Authority. Attempting to use ‘ldapsearch’ also returned errors about certificate validation. After some fumbling around I found that the clients need to be configured to handle such a circumstance. My solution for now was to make them trust Server1 as a Certificate Authority and also to enable an option in ldap.conf which affects how certificate requesting operates (and how strict to be).

Anyways more testing needs to be done but I think it’s working well now. Once I verify this I can configure Server2, and all of the clients too.

This afternoon I also configured Squid on Server1. It was pretty straight-forward. ‘sed’ made parsing through Odin’s configuration file easier. Now I have a barebone copy of squid.conf to use. Nobody liked squid.conf.default… Some strange behavior when testing Squid from my room. Will need to look into this further. But pretty rewarding to use it, because it was even using itself as a nameserver.

So apparently in BIND9 they introduced a feature known as “views”:

The view statement is a powerful new feature of BIND 9 that lets a name server answer a DNS query differently depending on who is asking. It is particularly useful for implementing split DNS setups without having to run multiple servers.

This is a useful feature in our case because I needed a way to implement fake DNS and real DNS without having to run DNS on a separate server. When I set up BIND on the two hallservers, I decided to use these “views.” Formerly, Vidar’s DHCP informed unregistered clients in the temporary (10.242.x.x) range that their DNS server is Vidar. Vidar resolved everything to itself, meaning they’re forced to go to the register pages when they’re trying to go to MySpace.

In the new setup, the hallservers will serve this DNS. When asked by 10.242.x.x clients, they will resolve everything to the host that ends up serving registration pages. When asked by registered clients, they will resolve as normal. Anyway here’s a snippet:

named.conf

acl authorized {
    66.165.0.0/19; // registered
    140.160.0.0/16; // academic
    10.242.0.0/19; // temporary
};

        allow-query { authorized; };
....

view "temporary_ips" {
  match-clients { 10.242.0.0/19; };
    recursion no;
        zone "." {
          type master;
          file "fake-dns/fake.root";
        };
  };

view "registered_ips" {
  match-clients { 66.165.0.0/19; 140.160.0.0/16; };
    recursion yes;
        zone "." {
          type hint;
          file "real-dns/real.root";
        };

Obviously fake.root will resolve all requests to the registration host.

I also set up Server1 and Server2 with SLAPD. All four boxes are also set up as LDAP clients. They use Server1 as a server. The office machines, running Gentoo, appear to interact well with Server1’s SLAPD…and all the FreeBSD boxes are running the latest nss_ldap (which broke things on Depts/Kulshan before).

The four 1U rackmount servers (Dell PESC1425) arrived today and they’re pretty awesome. I put FreeBSD onto one and left it running for if I can’t sleep again tonight. As requested, I came up with a semi-formal migration plan for the Hallservers. That is, “How are we planning to move these critical services to a pair of twin rackmounts which are redundant.” I talked with Nick about it and he made some clarifications and suggestions. Once the whole plan is finalized it will be pretty cool. I never cease to be amazed with how much I learn at this job.

Funny Nick says it’s interesting that I’m not paralyzed with fear at having to migrate every single server my first year here… maybe it just hasn’t caught up to me yet. Excitement > Fear…

Within the next few days I plan to actually make clean installs (very similar ones) on the new boxes. A few things will be the same among them all, for example the kernels, base system, some necessary ports, etc. Hopefully things go smoothly and I will be able to mold them into their respective roles. While the Hallservers are the “most important”, they’re also the most customized. I’m not sure if I want to start with those, or start with something simple like Odin migration (MySQL & Backups). Either way I’m excited to get at least one box migrated very soon, and follow with the rest. Tweaking can follow that. So much to do but so very rewarding!

Nick did indeed fool me today. But he’s starting a war he can’t win….

Alvis’ Apache build is really messed up after an ‘emerge uDav world’ from sometime during the beginning of summer (thanks, Gentoo). While being able to do things like ‘emerge world’ is kind of nice, it presents problems. I don’t think it is intelligent enough to always emerge dependencies in the proper order. So you’ll end up with binary incompatibilities, especially when dealing with Apache2 and such. Not only that, but Alvis is running PHP5, along with LDAP support, SSL, etc. and an unstable version of Apache2 also. I reemerged Apache2 tonight and all of the dependencies in the proper order. However, still experiencing same problems. Something about advancedFind.php is breaking it. It is easily replicable with certain names like “matt” for example, and a few others. Funny how memory corruption works like that. Very frustrating.

Finally activated a few iptables rules on Alvis and Kulshan today. No more unauthorized remote MySQL access.

The servers have arrived and need to be picked up. So begins a quest to migrate everything to new OS’es on new boxes.

SummerStart is underway. Nick and I worked at the booth today for a little while which was fun. Somehow this evening around 8:00pm my IP was unregistered. I’m really not sure how it happened. I have noticed that lately ipreg-summary.php has been “broken” for some reason. I looked through it today and there were some problems with incorrect casing. Hopefully it is now fixed and maybe I’ll be able to tell in 19 minutes when it tries again.

PacketShaper has been configured most of the way. I just need to transfer the old class definitions and such to the new one.

Servers are still in production.

The four new servers have been ordered. We did not go with the 2 GB RAM upgrade. We can beat the quote dell gave us elsewhere, and upgrading RAM is easy. This should be a very fun/interesting project, getting everything moved over and organized. I can’t wait until it’s all organized.

In other news, our Packetshaper model is, according to Right Systems, a 9500 ISP. I was told by the main contact “Everything I have tells me it’s a 9500 ISP.” I will have to find a serial cable and fire it up and check that way then. Packeteer said “It should say ISP on the unit.” Which it does not. Quite the mystery.

The drama with Cisco is interesting. It sounds like something from a movie. Michael Lynn, security expert, filed the papers to resign from his job at ISS, only 2 hours before giving a talk at the Black Hats Conference in which he explained that Cisco’s IOS (operating system used in their routers and such) is vulnerable to code execution (even using some of the old, known exploits).

More on that here: http://www.securityfocus.com/news/11263

Should be an interesting day when some proof-of-concepts regarding Lynn’s work are publicly accessible.