Following up on the dev box upgrade article, I’ll try to describe the process of blind luck and educated guesswork that enabled my network to support “fat clients”.
My dev box had a conflict. I needed memory bandwidth and processing power, but I didn’t need local storage. Ever since a near miss on a total loss of all my development work, I had moved everything onto a central server, which performs scheduled backups, has uninterruptible power and redundant disks. This meant the dev box only needed to store the OS and the software tools I used to do my thing. This was fine when I was using the slower Pentium IV … its udma 66 hard disk was plenty fast for the kind of bandwidth that machine had to deal with. Aside from the drive being noisy, all was well. After I upgraded the box, I first tried using the old drive again. It worked OK, but I could feel it was a real bottleneck, so I went with a modern sata drive instead. This eased the bottleneck some, and it was quiet. However, it was also wasteful, 90% of the local space was never going to get used. This waste put me in conflict as to how to resolve the situation, so instead of fixing the problem, I just ignored it.
Lately, I had been playing around with virtualization on my file server. It had also been upgraded recently, and was flush with diskspace and bandwidth. One of the fruits of my virtualization experimentation was Windows on Linux. Now I’m not talking emulation like Wine or Crossover or whatever else there is… I’m talking about a complete virtual machine living inside the memory of my server (which runs a flavor of Redhat Enterprise Linux called Centos). This gave me ideas, wild and crazy ideas! If I ran linux as the host os on my dev box, I could boot it off the network, and then run Windows XP as a guest, in a virtual machine!
The first steps were easy:
1) My network is ‘managed’ by a ‘lill router, the WRT54G, running some great firmware called DD-WRT. DD-WRT gives your $50 router $5000 worth of software features and a real nice interface to manage it all with. I configured my router, which was already playing the role of dhcp and local dns, to act as a bootp server as well.
2) Bootp works hand-in-hand with a file serving daemon called TFTP. The router can run tftp, but it’s storage is limited and I don’t feel like doing an SD Card Upgrade, so I gave the tftpd role to my network fileserver. After a quick “yum install tftpd” and an edit to “/etc/xinetd.d/tftp” I was all set.
3) Pxelinux.0 is a network boot image of sorts. It is built on syslinux, which is a microscopic distribution commonly used for booting heavier versions of linux. You see it used to boot almost every linux distro’s install cd and live cd. This special file is used to search out a configuration file on the tftp server that tells it where to find a kernel. It first tries various combinations of the client MAC address, followed by various other hexadecimal numbers and finally gives up and looks for a config named “default”. This lets you pass along different configurations to different computers or groups of computers.
4) Find a victim distribution. Ubuntu linux appears to be wildly popular right now. I’ve been running it for a few months on my laptop and I think it’s nice. One benefit of popularity is lots of people try lots of things, and some of them write about it on the interweb. So because there was a lot to read on diskless ubuntu, I stuck with that distribution. The recommended method is to install a local copy of the OS, mount your nfs root, and copy everything over. Instead, I just opened a new virtual machine on the server, installed ubuntu there, and then copied everything over. This allowed me to tweak and experiment to my heart’s content, without troubling a physical machine with lots of reboots and such. Three simple commands did the job; “mount -tnfs -onolock , “cp -ax /. /mnt/.” and “cp -ax /dev/. /mnt/dev/.”. I think the /dev part is redundant, since 2.6 uses devfs but I’m not a big guru on linux.
5) Copy that kernel and bring its ramdisk too. For whatever reason, the kernel needs a crutch to help it get going. On a disk based system, the crutch (ramdisk) might contain drivers to enable the kernel to work with a raid controller or a funky filesystem. On a diskless system, the ramdisk contains drivers for the network card and a basic dhcp client and tcp/ip stack. So bootp loads syslinux. Syslinux loads the ramdisk and the kernel, and setups up some things for the kernel. Then the kernel gets executed and sets up a bunch more stuff and then starts up init. Once init is up and running, things take off and the pretty soon you’re at the gdm greeter. You do need to rebuild the ramdisk, since the default is for a local disk boot. Changing one line in the initfs config file from boot=local to boot=nfs is all that’s needed, then just rerun the script that generates the initrd.
Now the job gets a little more interesting.
6) Configuration confusion. If you have a few systems sharing a single nfsroot, they’ll step on each others toes. I don’t plan to run a huge number of hosts, more than likely it’ll just be one. But, I wanted to learn some tricks anyway. First, the system hostname is very important to a lot of linux services. There didn’t appear to be an easy way to setup the hostname based on information from the DHCP server. I don’t know if there is a proper way to do this, and asking for help on a linux forum is generally an exercise in futility. Anyway, I came up with a script that init runs early on startup that sets the hostname from dhcp and dns information. It should work fine for any network that has local dns of its own IP addresses. The other config file to worry about is xorg.conf. Luckily, Xorg is smart, and reading the man page for xorg.conf, it shows that X will look in various places for various combinations of a config file name. The option that made the most sense for my setup was /usr/etc/X11/xorg.conf.hostname … so I nuked the original /etc/X11/xorg.conf and created a new one custom for each host (I’d been using a laptop to test this with before moving to the desktop). Now each host has its own X configuration specific for the display and video card. Lastly, the diskless howto’s recommend / recommend against mounting various directories on a ramdisk called tmpfs. I have some mounted that way, and others are shared. This will require more study if I use this knowledge in a bigger enviroment.
7) Get virtual. Installing VMWare Player was pretty straight forward… pretty much you just answer all the questions with the default “yes”. VMWare is a little buggy over nfs for some reason, and I need to sort that out since I can’t have the virtual machine crashing and failing to start. I built a windows xp pro virtual machine on my fileserver, which runs VMWare Server, and then I just open that machine on the dev box using vmplayer. I was happy to see Windows XP boot up just fine, and it actually saw the usb devices I had connected to the host machine. Drivers for my HP Printer installed and work great. Sound was broken at first, since vmplayer uses OSS, which my linux distro uses ALSA … there’s a quick and easy fix that involves loading an alsa-oss emulator library prior to starting vmware. Presto, sounds are mixed in just fine with the hosts sounds now.
8) Get busy. The main task of reading datasheets and drawing schematics I will do natively in Linux. Windows XP is there pretty much for running my pic programmer and compiler. The windows guest os mounts its smb shares off the file server as normal, and is actually pretty responsive.
Overall, this was a great project. I learned a great deal, of which I might be able to reuse for some of my educational customers. I haven’t done any concrete testing, but I’m pretty sure this diskless system is as fast or faster than a single drive system would be. The connection from client to server is over gigabit eithernet, and the server is running a multidrive RAID array. The last network benchmarks I ran yielded writes of 32meg/sec and reads of 60meg/sec.