ResourcesMark
Computers
- Lychee (serves /home)
- Guava
- Melon
- Grape
- Kiwi
- Compbio
- CentosNodes
- CentosWorkstation
- ThinClients
- Mitchell Workstations
- Cisco Switches
Other Information
What to do when the computers crash and an admin is not there
- Shutdown workstations, nodes, lychee and guava
- Turn on:
- guava
- lychee
- nodes
- workstations
How do I find out how to use a program
- Use the manual pages. Run: man PROGRAM_NAME
How do I transfer files from lychee without slowing down the network
- Use rsync with the --bwlimit option. Read up on using rsync in the manual pages
How to enable a computer to send automatic emails to non-UQ addresses?
- If postfix is used for SMTP, in /etc/postfix/main.cf make sure relayhost = smtp.uq.edu.au
What kinds of workstations are used in the lab?
- AMD 64-bit workstations, for floating-point consistency with the cluster, which also has AMD 64-bit processors. We also have only NVIDIA video cards for ease of maintenance.
How to reinstall/install an NVIDIA driver?
- An NVIDIA display driver depends on a specific version of the kernel, so if the kernel is updated it will not work any more (you will just get a blank screen). To install an NVIDIA display driver from the .run file downloaded from nvidia.com (e.g. cudadriver_2.3_linux_64_190.18.run - 190.18 is the driver version; linux_64 means it can be installed for any version of a 64-bit linux kernel), do the following as a user:
- Login via ssh
- mv /path/to/cudadriver_2.3_linux_64_190.18.run /tmp # this step ensures root can read the run file because its original location, e.g. in a nfs mounted directory, may not be readable by root.
- sudo init 3 # this closes the X server
- sudo sh /tmp/cudadriver_2.3_linux_64_190.18.run # this starts the driver
- Say yes to everything
- sudo init 5 # this restarts the X server
How to set up a VPN connection from a workstation?
- Example workstation: uqmd00.md.smms.uq.edu.au
- Example username: bob
- Example vpn server/Host: a.com, at 12.13.14.15, on network 12.13.14.00/255.255.255.0
- Example server of interest only accessible through vpn, a.a.com at 12.13.16.1, on network 12.13.16.0/255.255.255.0
- The Host, GroupName and GroupPwd can be read from CISCO .pcf file, if only enc_GroupPwd is found you can decrypt it using this site.
- Get ITS to allow incoming Cisco VPN connections from a.com to uqmd00.md.smms.uq.edu.au
- Once these are done, follow the following instructions to set up VPN:
- sudo yum install vpnc # Install vpnc
- sudo vim /etc/vpnc/vpnc.conf # Edit settings
IPSec gateway a.com IPSec ID GroupName IPSec secret GroupPwd
- sudo /sbin/vpnc /etc/vpnc/vpnc.conf # Start vpn client. Warning: this screws up the routing and DNS settings. The following steps are to correct them.
- sudo /sbin/route # Print route table to check status. All internet now goes via the vpn (tun0).
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface a.com gateway-b195-12 255.255.255.255 UGH 0 0 0 eth0 152.98.195.128 * 255.255.255.192 U 0 0 0 eth0 12.13.14.00 * 255.255.255.0 U 0 0 0 tun0 default * 0.0.0.0 U 0 0 0 tun0
- sudo /sbin/route add -net 12.13.16.0 netmask 255.255.255.0 dev tun0 # Add route so traffic to a.a.com goes through tun0
- sudo /sbin/route del default dev tun0 # Remove wrong default gateway. Warning: Internet traffic doesn't get routed anywhere
- sudo /sbin/route add default gw 152.98.195.129 eth0 # Put right default gateway in. Internet traffic goes through the normal gateway for the MD VLAN (gateway-b195-128 = 152.98.195.129)
- sudo /usr/bin/vim /etc/resolv.conf # Set DNS settings
domain md.smms.uq.edu.au search md.smms.uq.edu.au nameserver 130.102.128.43 nameserver 130.102.2.15
Endnote X on kiwi
- To set up Endnote X on kiwi for a new user, the files EN10Cwyw.dot, EN10Cwyw.WordXP.wll and ENWebCwyw.dot must be copied to the USERNAME's "C:\Documents and Settings\USERNAME\Application Data\Microsoft\Word\STARTUP" directory from "C:\Program Files\EndNote X".
Linux introduction for new users
- Please read Introduction_To_Computing
System Configuration (workstations, grape)
Thin Client/workstation configuration
System Configuration (lychee & mango*)
lychee system
System Configuration (Melon - Solaris ZFS)
NFS disk layout
- lychee:
/home 500G (for MDGroup/BMMG, quota for each user: soft 30G, hard 40G)
/home1 250G (for SBeatson)
/home2 250G (for KobeLab)
/data5 500G
/data7 150G (backups of old user accounts)
- guava:
/data1 500G
/data2 500G
/data3 500G
/data6 180G
- grape:
/data8 400G
/data9 400G
/data10 400G
- melon:
/melon1 8.1T (data pool for MDGroup/BMMG, zfs quota for user 500G)
/melon2 8.1T (data pool for SBeatson)
Cluster Command & Control (c3) tool suite
c3 is install on lychee, the configure file is /etc/c3.conf . The binaries are installed in /opt/c3-4 .
To run a command to all nodes:
sudo cexec <command>
To upload a file to all nodes:
sudo cpush <file> /path/for/the/file
Alarm handling/Shutting down the rack
Booting up the system
1. Boot up NFS servers: guava, grape, melon. melon may take a long time to boot, so be patient.
2. Boot up our main server: lychee
3. Boot up all cluster nodes and all workstations.
System Configuration (windows and Linux laptop)
System Configuration (CCMS Blackhole)
Grape queueing system
Add new user (LDAP)
Please use the system "firstnamel", where l is the first letter of the last name, for new user names
Managing Fedora-DS
To get a untaken UID for the new accont, use nextUID.py placed in '/opt/scripts'. Note that the UID may be already in use on grape.
To add a group of new users with similar uid, use script ldapBatchAddUser.py placed in '/opt/scripts/'. You will have to run it as root, and keep in mind the following things.
This program helps to create multiple user accounts on the fly.
In order to use it, you have to provide some basic information
of the new accounts. The new accounts will be named as "BASE$N",
in which "BASE" is the base name, and $N is the count. You have
to specify the starting number of $N. Users' home directory
will be /home/BASE/BASE$N, and bear in mind that this program
does not create /home/BASE automatically. The program will invoke
"nextUID.py" to set UIDs for the new accounts, However, you still
have to take care of the GID issue yourself. All new accounts
will use "BASE" as their passwords.
Modify the program yourself to make it better suit your need.
Use CTRL+C to interrupt the program when typing something wrong.
add user for kiwi
- go to administration tools in control panel, and create a user with an identical name as the one in LDAP.
- right click and set up properties of the user the same as the others.
- right click and set the passwd of the user.
- go to start menu -> Services for Unix administration, select "user name mapping" on the left panel, select maps on the right, then add the new mapping. If the user id does not appear in the unix user list: on lychee, do
getent passwd > /nfs4exports/passwd.txt
getent group > /nfs4exports/group.txt
copy those files to kiwi's c:/ , and refresh the unix user list.
Add new user (grape)
Create account in /etc/passwd
useradd -u NEWUID -g md NAME
Update NIS (for cluster authentication)
cd /var/yp/ make
Gromos Live (ubuntu based live DVD)
The latest DVD image can be found in /home/exercise/LiveDVD
The DVD is based on the one from IGC ETH with some minor modifications including:
- changing gnome background
- removing unnecessary language and game packages
- changing firefox homepage (/usr/share/ubuntu-artwork/home/index.html)
- changing license file in GROMOS directory, and moving GROMOS directory from /opt to /usr/local
- installing gromacs to /opt
- removing nfs-common, portmap and games packages
Modify the Live DVD
Guidelines on how to modify an ubuntu live cd can be found in the following urls. Note that ALL of the above modifications are made in squash file:
http://www.debuntu.org/how-to-customize-your-ubuntu-live-cd
http://www.atworkonline.it/~bibe/ubuntu/custom-livecd.htm
Here is a rough walkthrough:
- mount the original iso and unpack everything to a folder, here we choose /media/live as our working directory.
sudo mount -t iso9660 -o loop /path/to/gromos-live.iso /mnt/dvd sudo cp -a /mnt/dvd /media/live/dvd
- mount the squashfs image and unpack everything
sudo mount -t squashfs -o loop /media/live/dvd/casper/filesystem.squashfs \
/mnt/squash
sudo cp -a /mnt/squash /media/live/squash
- umount the original iso
sudo umount /mnt/squash; sudo umount /mnt/dvd
- chroot to the squash folder and start to make modifications
cd /media/live/squash cp /etc/resolv.conf etc/ sudo chroot . mount -t proc none /proc mount -t sysfs none /sys export $HOME=/root
- make changes to the files in squash
- to install new package or remove a package, use apt-get as in a normal ubuntu installation.
- files can be put directly in to the directories and then accessed after boot, but make sure files in /home/ubuntu have owner and group id set to 999:999
- clean up before reimaging
rm -rf /tmp/* apt-get clean umount /sys umount /proc rm /etc/resolv.conf exit
- regenerate squash image
cd /media/live/dvd
sudo chroot ../squash dpkg-query -W --showformat='${Package} ${Version}\n' \
> casper/filesystem.manifest
sudo rm casper/filesystem.squashfs
sudo mksquashfs ../squash casper/filesystem.squashfs
- regenerate iso image
cd /media/live/dvd
sudo find . -type f -print0 |xargs -0 md5sum |sudo tee md5sum.txt
cd ..
sudo mkisofs \
-o Gromos-Live.iso \
-b isolinux/isolinux.bin \
-c isolinux/boot.cat \
-no-emul-boot \
-boot-load-size 4 \
-boot-info-table \
-r \
-V "Gromos Live" \
-cache-inodes \
-J \
-l \
dvd
Port the live DVD to USB key drive
please refer to the following url. As Gromos Live is based on Ubuntu 7.10, we cannot use scripts written for other versions.
http://www.pendrivelinux.com/2007/09/28/usb-ubuntu-710-gutsy-gibbon-install/
Install the live DVD to a hard disk
- copy the unpacked /casper to the partition that you want to install on.
- run sudo grub-install --root-directory=/media/hda1 --force-lba --no-floppy /dev/hda to install grub to the hard drive.
- write the following content to /boot/grub/menu.lst, and modify it if needed.
timeout 10
color cyan/blue white/blue
title Start Ubuntu in Persistent mode
root (hd0,0)
kernel /casper/vmlinuz boot=casper initrd=/casper/initrd.gz
ramdisk_size=1048576 root=/dev/ram rw persistent
initrd /casper/initrd.gz
boot
title Live - Start or install Ubuntu
root (hd0,0)
kernel /casper/vmlinuz boot=casper initrd=/casper/initrd.gz
ramdisk_size=1048576 root=/dev/ram rw
initrd /casper/initrd.gz
boot
title Start Ubuntu in safe graphics mode
root (hd0,0)
kernel /casper/vmlinuz boot=casper xforcevesa initrd=/casper/initrd.gz
ramdisk_size=1048576 root=/dev/ram rw
initrd /casper/initrd.gz
boot
title Check CD for defects
root (hd0,0)
kernel /casper/vmlinuz boot=casper integrity-check initrd=/casper/initrd.gz
ramdisk_size=1048576 root=/dev/ram rw
initrd /casper/initrd.gz
boot
title Memory test
root (hd0,0)
kernel /install/mt86plus
boot
How to compile GROMOS96 on APAC (ac)
In order to run parallel version of GROMOS96 on APAC (ac) the following changes should be made:
In ../src/promd_db.F change the line
call system (
$'cat /proc/cpuinfo |grep MHz| awk {print $4} > tmp.info.xyz')
to
call system (
$"cat /proc/cpuinfo |grep MHz| awk '{print $4}'> tmp.info.xyz")
Single quotes became a double quote and the pair of single quotes before and after the {} became just one single quote each.
You should make the same corrections a couple of lines later as well even though the code will not enter that conditional.
Then a bit further on change it to read
call system (
$"cat /proc/cpuinfo |grep model |
$ awk -F: '{print $2}' > tmp.info.xyz")
We should thank Dr Margaret Kahn from APAC for this solution.
Revision Control System (git+gitosis+redmine)
Git is used as revision control system on lychee, as it makes merging branches easier. Gitosis is installed to provide a safer and easier way of publishing, and home directory of user "git" is /srv/git, in which all the source code is put. Redmine provides a friendly user interface for project management and code browsing, and can be accessed at http://lychee:3000 .
Configuration of Gitosis
Gitosis uses pure git commands to update its configurations.
To start, first add yourself to the admin group of gitosis.
sudo -H -u gitosis gitosis-init < /path/to/your/sshkey/id_rsa.pub
Then you can download its configurations
git clone git@lychee:gitosis-admin.git
modify gitosis.conf, to add a new project named "atb"
[group atb] writable = atb members = roy root alpesh pramod
copy user roy, root, alpesh, pramod's id_rsa.pub to {{{keydir}}}, name them to <uid>.pub
Then push the updated version
git commit -a -m "project atb added" git push
Configuration of redmine
- install rubygems, and use it to install right version of rails according to redmine's homepage.
- copy config/database.yml.example to config/database.yml and edit it to reflect the type of database in use. Here we use sqlite3.
- run
rake db:migrate RAILS_ENV="production"
rake redmine:load_default_data RAILS_ENV="production"
- now the page can be served by running
ruby script/server webrick -e production
default login/passwd : admin/admin
- add {{{ruby script/server webrick -e production}}} to /etc/rc.local to make a daemon
git user configuration
To make git aware of who is the contributor and whom to blame, add the following to ~/.gitconfig, email address is important
[user]
name = <full name of the user>
email = <email of the user>
clone the repository:
git clone git@lychee:atb
Tips
Use rsync
An efficient way to transfer files or directory between computers:
rsync -avzP --delete -e 'ssh -p20' username@old_server:/old_place/ \
username@new_server:/new_place/
Important, the flag --delete will delete any file at the target which does not appear at the source. Also you should put at the port flag the port on our system (if you don't know, ask the one of the 'Guru's).
Submit jobs from any client machines
- You need to setup key based ssh login.
- Put the following in your .bashrc, then you can use qsub, qdel, showq, qnodes, qstat from everywhere.
if [ "$HOSTNAME" != "lychee" ]; then
function qsub()
{
ssh lychee "cd $PWD;
/opt/bin/qsub $1";
}
for i in showq qdel qstat qnodes showstart; do
alias $i="ssh lychee /opt/bin/$i"
done
fi
Mount your home directories on your linux laptops with sshfs
First you have to install fuse, and
modeprobe fuse
to install the module. You may also want to add fuse to your /etc/modules.
Then, use the following line to mount your sshfs. You must also have public key based authentication setup correctly.
sshfs -p 2023 <account>@lychee: /path/to/your/local/mount/point
For mac users, there is a sshfs port for mac users.
How to compile GROMOSXX 0.3.1 (Oct09) on NCI(APAC) Vayu and Xe clusters with openmpi
Get the source code from: /marksw/gromos.ethz.ch_repos_gromosXXc++/Oct09/trunk/
module unload intel-cc module unload intel-fc module load gsl/1.12 module load fftw3/3.2.2
cd gromosXX ./configure --prefix='give the path name' --enable-openmp make make install
How to compile GROMOS++ 0.3.1 (Oct09) on NCI(APAC) Vayu and Xe clusters
Get the source code from: /marksw/gromos.ethz.ch_repos_gromos++/latest/trunk
module load gcc/4.3.3 module load gsl/1.12
cd gromos++ ./Config.sh ./configure --prefix='give the path name' --enable-shared=No CC=gcc CXX=g++ make make install
How to compile GROMOS96 (24June04) on NCI(APAC) Vayu and Xe clusters
get the tarball from /marksw/gromos96/24Jun04/SOURCE/gromos96_24Jun04.tar
tar xvf gromos96_24Jun04.tar cd gromos96_24Jun04
Edit the config.dat file to select the Intel Fortran compiler
OS = LINUX COMPILER = IFC
make gromos to build the entire distribution & cleanup make distrib to build the entire distribution make promd to build several versions of promd only make promdopt to build optimized versions of promd only
How to compile GROMOSXX on NCI(APAC) Vayu and Xe clusters
Make sure that lam is properly installed. To install your own version, download it from the web and configure it like this:
./configure --prefix=$PATH_TO_LAM/lam/install
Make sure that you have fftw3mpi installed. You can download it from the web and compile it yourself. Configure like this:
./configure --enable-mpi CXX=$PATH_TO_LAM/lam/install/bin/mpiCC CC=$PATH_TO_LAM/lam/install/bin/mpicc F77=$PATH_TO_LAM/lam/install/bin/mpif77 --prefix=$PATH_TO_FFTW/lib/fftw
Now you can install md++. Configure like this (revisions that need fftw, i.e. the ones that have p3m):
../configure --with-gsl=$GSL/lib/gsl --disable-debug --disable-shared --enable-mpi --with-fftw=$LIB/lib/fftw/ CC=$LAM/lam/install/bin/mpicc CXX=$LAM/lam/install/bin/mpiCC
