Cloud At Sara
SARA in the CLOUD
by Pjotr Prins, Wageningen University, Dept. of Nematology
The super computing center SARA in Amsterdam has a pilot in CLOUD computing in which I participate as a tester. Here I quickly document my experiences over november 2009.
The setup consists of five dual-quad CPU's - i.e. 40 cores (additional to Nematology/Bioinformatics 24 cores' in my 'CLOUD') available for some hard hitting on top of Ubuntu with KVM. SARA supplies a few images and documentation.
The login to the control server (named ui) is via:
ssh -XC firstname.lastname@example.org ssh -XC email@example.com
Easiest to provide a tunnel, say
ssh -L 20201:ui.claudia.sara.nl:22 -f -N firstname.lastname@example.org
And now you can ssh and scp etc. directly through
ssh -p 20201 user@localhost
easiest to setup a key too. On your local machine:
ssh-keygen -t dsa
and add the content of ~/.ssh/id_dsa.pub to the ~/.ssh/authorized_keys file on UI (and later any images) as described here. Using ssh keys allows you to login without typing passwords every time.
A number of commands are listed on the CLAUDIA wiki listed above. Additionally
for x in 0 1 2 3 4 5 ; do echo -n $x :; onehost show $x | grep USEDCPU ; done
0 :USEDCPU=0.799999999999955 1 :USEDCPU=0.799999999999955 2 :USEDCPU=0.799999999999955 3 :USEDCPU=0.799999999999955 4 :USEDCPU=3.20000000000005 5 :USEDCPU=1.60000000000002
You can see the CLOUD is not busy. Currently there are two types of images running:
user@ui:~$ onevm list ID NAME STAT CPU MEM HOSTNAME TIME 193 vm_cloud runn 0 4194304 host05 05 21:02:00 194 ubuntu-9 runn 0 2097152 host01 02 23:07:56 195 ubuntu-9 runn 0 2097152 host02 02 02:38:10 200 debian runn 0 2097152 host03 01 19:36:52 201 debian runn 0 2097152 host06 01 19:14:56 202 ubuntu-9 runn 0 2097152 host04 01 17:40:53 203 ubuntu-9 runn 0 2097152 host06 01 17:40:52 204 ubuntu-9 runn 0 2097152 host05 01 17:40:50
Starting an image
Current images are:
user@ui:~$ ls /data/images/ debian-503-i386-CD-1.iso scilin5.3-10G.img ubuntu-9.10-10G.img debian-lenny-10G.img ttylinux.img ubuntu-9.10-server-i386.iso
These are 32-bit images. A 64-bit image I have to make myself - will do that a bit later. First and foremost I am a Debian-guy, so the first bit is easy. Copy the image to use your own - you don't want to update a shared image:
cp /data/images/debian-lenny-10G.img ~
copy and edit the template to match:
cp /data/templates/debian-lenny-10G.template ~ vi ~/debian-lenny-10G.template
First I rename my image
cp debian-lenny-10G.img debian-lenny-default-32.img cp debian-lenny-10G.template debian-lenny-default-32.template
I am greedy (needlessly to say) so I grab all CPU's (8) and a little more RAM. The diff is
< CPU = 1 < MEMORY = 2048 --- > CPU = 8 > MEMORY = 4096 < source = "/data/images/debian-lenny-10G.img", --- > source = "/home/cloud10/debian-lenny-default32.img",
onevm create debian-lenny-default-32.template onevm list 205 cloud10 debian fail 0 0 host03 00 00:00:09
not good. I had used the wrong name for source. The log can be found in
Try again and it reads:
206 cloud10 debian runn 0 0 host03 00 00:00:42
and, yes! it is running on host03. If it is in a pending state, kick it to a node with
onevm deploy 206 host03
Now you know the ID you can get extra info with
onvm show 206
which gives the IP etc. It may take a while to boot, but then
ssh email@example.com Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Tue Nov 3 06:17:28 2009 cloud@debian:~$
That looks good. 'top' shows 4 Gb of RAM, 'df' shows I have 8.5 Gb of disk space. Only problem is I have one CPU I see with cat /proc/cpuinfo! So deleting the previous VM
onevm delete 206
and recreate, now with 4 CPUs. For this I read some of the Open Nebula documentation, as the number of CPU's did not seem to matter in the configuration. The configuration of onevm (part of the open nebula project, see link below) is in /etc/one/oned.conf which shows in particular vmm_kvm/vmm_kvm.conf and im_kvm/im_kvm.conf are relevant. The first contains the maximum of (virtual) CPU's used. This has not been set, so maybe it defaults to one? Inside the Debian image:
cloud@debian:~$ dmesg|grep -i cpu [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] kvm-clock: cpu 0, msr 0:3baf81, boot clock [ 0.000000] SMP: Allowing 8 CPUs, 7 hotplug CPUs [ 0.000000] PERCPU: Allocating 37960 bytes of per cpu data [ 0.000000] NR_CPUS: 8, nr_cpu_ids: 8
Ah! The image sees 8 CPU's, but only uses one. This is hotplugging CPU's, a feature of the Linux Kernel. You can switch them on with
My help desk Floris Sluiter, of SARA, suggests:
To enable multicore machines you can add "vCPU = n" to your template: NAME = debian CPU = 2 # NOTE: CPU and vCPU should be the same! vCPU = 2 MEMORY = 2048
NOTE: the number of cores after CPU and vCPU should be the same (otherwise you risk interfering with other users). This is because CPU is used by the scheduler and vCPU is used by the image.
Restarting the VM takes a while, so I tail it with
onevm list tail -f /var/log/one/218.log
and yes, now I have all my CPU's listed in 'top'!
Securing the image
Time to start securing my image. Just by adding a user:
sudo bash adduser yourname passwd root (something very safe) passwd cloud (something safe)
next I edit /etc/ssh/sshd_config to disable root logins by adding
change the entry in sudo from cloud into yourname
that means to gain root I have login as a different user. This is standard protocol, for me. To really secure your system you may want to consider disabling password logins (use ssh keys instead), and perhaps limit the number of originating IP addresses. But anyway, that is beyond this document, and the security really stands or falls with the security of the KVM host system (which is controlled by the CLOUD facility at SARA), which is beyond my control. Anyone can take down my VM and mount the image on the local disk system, leaving it open to compromise. For now this is prevents immediate strangers from entering your VM.
One nice aspect is that we can clone this image later. So all the actions we have on this 'master' image won't have to be repeated for the others.
Backup the image
At this point you may want to clone your updated disk image (just copy it to something else). This can be done at any point in time. Do stop the VM first - otherwise there is a possibility the image file system is not consistent. Cloning a live VM is possibible, but that is something else.
Unfortunately in the current setup at SARA OpenNebula does not cleanly copy back the image. The workaround is to *stop* the running VM and copy the image located at /home/oneadmin/ID. So
onvm stop 218 (wait a bit until onvm list shows 'stop') cp -vau /home/one/218/images/disk.0 save.img onevm resume 218
I tested this saved image. It works.
Next step is to install software. On Debian I have a list of packages I normally install/update, for example:
apt-get update apt-get install less openssh-server openssh-client \ vim bzip2 ruby perl \ screen unzip \ sudo gnupg locales lynx \ mc rsync python make astyle gcc ncurses-bin \ unzip tree git-core tzdata psmisc
With Debian you may want to
dpkg-reconfigure locales dpkg-reconfigure tzdata
Once all this is done, make a backup of your image. You don't want to repeat this work.
Adding a drive
I am going to build Nix package (see nixos.org). For several reasons, but mostly because I like reproducible systems. One tool I'll need is subversion, so
apt-get install subversion
Now visit my nix_on_debian writeup (in preparation)
apt-get install nfs-kernel-server nfs-common