Butlin:Unix for Bioinformatics - basic tutorial

This tutorial is still under construction. Please come back.

---

---

Before you jump into this tutorial

• Your command line prompt will end with a $ sign. So a $ sign in this tutorial tells you to type the stuff that comes after the $ sign into your command line.

• The words folder and directory mean the same thing. So I use them interchangeably.

• Linux is Unix re-coded under an open-source licence, the same way as R is a re-coded version of S. Here, when I use the term Unix, I refer to all Unix-like computing environments, i. e. the original Unix that comes with Macs as well as most Linux flavours.

• The practical part of the workshop will be done on the computer cluster of the University of Sheffield called Iceberg. It has scientific Linux installed. You will log into your own accounts on Iceberg.

• If you have never used a Unix command line, we suggest you concentrate on the boldly printed text in the first two sections in this module. Make sure you get through them in time so that you are fit to do the rest of the workshop.

• There are still small (hopefully not large) bugs lurking in this protocol. Please help improve it by adding comments.

Iceberg - making first contact

Access with the programme PuTTY

If PuTTY can’t be found on your computer, go to section 1.2 Access via an internet browser.

Chose “iceberg” and press “Open”.

Type in the login name that you have been given, hit Enter, then type in the password and hit Enter.

Access via an internet browser

If PuTTY is not installed, you have to use an internet browser to access Iceberg.

Follow the link “Connect to Iceberg now!” on this site.

Insert the username/login name and password that you have been given.

Under Iceberg Applications select Iceberg terminal.

A new window will pop up.

Iceberg access for Mac and Linux users

Open a terminal, then type at the command line prompt:

ssh   -X   your_username@iceberg.shef.ac.uk

You will be asked for your iceberg password and at the first time it usually issues a warning about accessing an untrusted server. Just confirm that you want to add iceberg to your trusted server connections. The -X switch opens a connection with X11 forwarding. If you don’t intend to open a GUI on iceberg, skip that switch.

Some basics

On the head node called “iceberg1” start a new session on one of the worker nodes by typing:

qrsh

No work should ever be done on the head node ‘iceberg1’ or else others and ultimately you will suffer the consequences :-)

If you’ve logged in via the web browser and used qsh (instead of qrsh) to start a new session, then you could open a programme that uses a GUI, e. g. firefox:

firefox

Where am I in the file system?

pwd

Here is a visual representation of a Unix file system:

taken from the Unix Tutorial for Beginners

Every Unix operating system has a root folder simply called /. Let’s see what’s in it:

ls /

List the files in the current directory, i. e. your home directory:

ls

Your home directory is still empty, or is it?

ls -a

The -a switch makes ls show hidden files, which start with a dot in their file name. Let’s create a new directory and a new empty file.

mkdir NGS_workshop
ls
cd NGS_workshop
ls
touch test
ls -l

The list output of ls prints out a lot of information about each file and directory.

drwxr-xr-x    4 cliff    user        1024 Jun 18 09:40 directory_name
-rw-r--r--    1 cliff    user      767392 Jun  6 14:28 file_name
^ ^  ^  ^     ^   ^       ^           ^      ^    ^      ^
| |  |  |     |   |       |           |      |    |      |  
| |  |  |     | owner   group       size   date  time    name 
| |  |  |     number of links to file or directory contents
| |  |  permissions for world
| |  permissions for members of group
| permissions for owner of file: r = read, w = write, x = execute -=no permission
type of file: - = normal file, d=directory, l = symbolic link, and others...

$ ls -lF

Note the forward slash at the end of file names, when you use the -F option. This indicates a directory.

$ ls -lh

How can I look up the manual for the ls command and most other Unix commands?

$ man ls

Navigation in the manual which is opened by the text viewer less:

Keyboard	What it does
f or Space	one screen size down
b	one screen size up
d	half a screen size down
u	half a screen size up
G	jump to end of file
g	jump to beginning of file
h	get help
q	exit help
q	exit less

Save yourself some typing:

$ alias ll=’ls -lFh’
$ man alias

However, this neat little shortcut is only active in the current terminal window. In order to create this alias each time you login into iceberg, add the above alias command line to your .bash_profile file:

$ nano .bash_profile

The dot at the beginning is part of the file name, so don’t forget it. Files whose names start with a dot are hidden files. At the bottom of the file add:

alias ll=’ls -lFh’

hit Ctrl+O and Enter on your keyboard to save changes, then Ctrl+X to exit nano.

$ source .bash_profile
$ ll

What is my file storage quota on iceberg?

$ quota

Exit iceberg by typing:

$ exit

… to quit the interactive session and get back to the head node and:

$ logout

… to log off the cluster.

Gearing up for work with files and directories

Log back into iceberg and start an interactive session with qsh. Create a new directory in the directory NGS_workshop:

$ cd ~/NGS_workshop

Note this is equivalent to:

$ cd /home/your_username/NGS_workshop

$ mkdir output
$ ll
$ ll output

Change into the new directory:

$ cd output

Note how your command line prompt has changed.

$ ll

Create five new empty files:

$ touch test test1 test12 test123 Test This_is_a_really_long_file_name_isnt_it
$ ll

Note: Spaces are important for Unix to parse the command line (but there is no difference between one and many spaces). So replace them with underscores in your file names. Generally, you can safely use the characters [a-zA-Z0-9._] in your file names.

Bash (short for Bourne Again Shell), the programme that provides the command line interface to Unix that you are currently using, comes with so-called wildcards:

$ ll test*

Two things to note here: 1) The asterisk stands for anything, including nothing and 2) Unix is case sensitive.

$ ll test?
$ ll *_*

Copy, move and remove files:

$ cp test* ..
$ ll ..

“..” stands for the parent directory.

$ ll ../../../..
$ cd ..
$ ll
$ rm test

Note, the rm command deletes the file (and with the -r switch also directories). It doesn’t move them into a “trash can”, in case you have second thoughts. It also, by default, doesn’t ask you for confirmation.

$ cp output/test* .

Note the dot at the end of the last command line. It’s short for the current working directory or “here”.

$ ll
$ cd output
$ echo haha

echo is bash’s print command.

$ echo haha > test1
$ echo hihi > test2

The redirection operator > redirects the output of the echo command into the file test1. Otherwise, echo prints to STDOUT, which is the terminal screen.

$ cat test1 test2
$ echo hohoho >> test1
$ cat test1

The >> operator appends the output of echo to the end of the file test1.

$ echo “hoohooo” > test1
$ cat test1

Redirecting the output of a command into an existing file overwrites it without notice. Remember this and be careful!

$ cp ../test12 test1
   

We are copying the file test12 from the parent directory into the current directory and save it as test1. Let’s see what’s in test1 now.

$ cat test1

?!?!?!?!?!? The cp command has overwritten the test1 file in the current directory with the content of the file test12 from the parental directory. This file was empty.

$ cat test2
$ mv ../test123 test2
$ cat test2

The mv command (which does cut and paste) has just done the same as the cp command. It has silently overwritten the file test2 in the current directory with the empty file test123.

Ok, you should be sufficiently scared by now. Here’s how you make these three commands safer: All three commands have a switch that causes them to prompt the user for confirmation before overwriting an existing file. It’s -i for all three commands. Check with:

$ man rm
$ man cp
$ man mv