Moving in

2009-06-07T20:40:00.000-07:00

So the topic of the summer is how to setup a cluster of machines and get them to talk to each other with the purpose of solving some difficult computation.

This is very interesting from theoretical CS point of view (what can be parallelized and what can't), but I don't think I am qualified to speak about that.

I am more of a practical guy.
Today I will tell you how far I have come so far.

HARDWARE:
The department I work for purchased 9 servers recently. They are IBM machines. U1 Rack mounted dual quad core servers. One of the machines is the "head node" which has also a storage array attached to it.

OS INSTALL:
We run an all debian operation at work and the debian of 2009 is "lenny" so I went down to the server room with an install CD and a couple of minues later I had the machines installed. If you have several monitors you can parallelize the install process.
Debian, by the way, has gotten REALLY good on the installer part. While installing the network card the installer realized that it was missing some firmware to get it going. It prompts the user "do you want to load firmware from removable media?".
If you say yes and there is a USB stick with the downloaded .deb firmware package the installer will load it up and continue.
This is the kind of installation functionality I enjoy. No more going through dmesg ...

trick: to minimize the time spent in the noise and cold-hot air currents of the server room you can install just the base system + package:openssh-server before the reboot. Press Alt+F2 to get to a new shell. Then say:


chroot /target

then you just run apt-get install openssh-server
and then you can come back to the first console and continue with the reboot...
the rest can be done from the comfort of your office / home

OS SETUP:
I will have to post a longer description about /etc.
To say something about the packages: pretty lean install with some X library that was needed for the MATLAB graphical installer and sun-java-6.

MATLAB:
There is the Matlab Distributed Computing Server installed on each of the nodes as well as the master node. There are two components: the mdce "service" that runs on all machines all the time and "jobmanager" and "worker" processes. You are meant to start one jobmngr on the head node and workers on the nodes.
I had some glitches with the licensing, but I got that sorted out.
I installed everything (license server on master node, network license for nodes) over a couple of days and did some tests.
The client side is handled by the parallel computing toolbox which has some GUI tools for connecting to the cluster.
After that you have to do stuff like. job =jm.createJob; job.createTask(@function,nargs,{args}); submit(job); waitForState(job,'finished'); data=getAllOutputArguments(job);

Now what remains to be done is to figure out which ports need to be open between the machine on which the jobmanager is running and the client machines.

After that the "job" is done and I get to go on vacation.

Cluster Admin

Moving in