Difference between revisions of "Computer lab introduction"

From MolEvol
(WiFi Login)
(Downloading files)
 
(223 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 
''"On the other side of the screen, it all looks so easy."'' (TRON, 1982)
 
''"On the other side of the screen, it all looks so easy."'' (TRON, 1982)
 +
 +
==Goals==
 +
By the end of this introduction, you should be able to:
 +
* Log in to the class cluster.
 +
* Navigate and understand the directory structure.
 +
* Create and edit files and directories on the cluster.
 +
* Move files to the cluster.
 +
Ask for help if any of these things aren't working by the end!
 +
 +
==Slack group==
 +
Slack is a messaging/discussion program that allows for large groups to interact as needed.<br/>
 +
There is a slack group set up for people to ask questions and to have discussions.<br/>
 +
The slack team is '''MBL-MOLE-2019'''<br/>
 +
You can use the Slack app on your laptop and your phone. <br/>
 +
If you can't join through the app/website, you can click on [https://join.slack.com/t/mbl-mole-2019/shared_invite/enQtNzA4NDY2NzQ3MzMzLTBhNWZiNzMzZTg4ZGI0NTVhN2MxMzYyNzY5ZDJhYTYzZWIyNDk1M2Q1OTA3ODgxNGZmNDJmODY5MDViODgzNDc this link]  (note that until Sunday, 28 July 2019, this link was for the 2018 version of the course, but it is now fixed).
  
 
==WiFi Login==
 
==WiFi Login==
To log on to the MBL wireless choose the MBL-REGISTER from the wireless list.
+
To connect to the cluster you must:
 +
 
 +
1. be logged on to the MBL-REGISTER wifi network.
 +
 
 +
2. have any virtual private network (VPN) turned off.
 +
 
 +
Full instructions can be found [https://intranet.mbl.edu/display/IN/MBL-REGISTERED here]
 +
 
 +
When you open a new browser window after you've registered, you should see a prompt for name and password.
 
Your username is your initials followed by the 5 digit number on the side of your identification card. Your password is the same.
 
Your username is your initials followed by the 5 digit number on the side of your identification card. Your password is the same.
E.g. if you name is Joe Bloggs and the your card has the number 12345 on the side then your login details are:
+
E.g. if your name is Jane Bloggs and your card has the number 12345 on the side then your login details are:
  
 
username: jb12345<br>
 
username: jb12345<br>
 
password: jb12345
 
password: jb12345
  
Some people have been having trouble, if it isn't working find or email Emily Jane ejmctavish@utexas.edu
+
Some people have been having trouble, if it isn't working find or email George (george.tiley@gmail.com) or any TA, or ask on the Slack group.
 
 
== Remote computer access software ==
 
We will use Secure Shell (SSH) and sFTP to connect to the servers.
 
The servers are powerful computers where we can run the programs much faster than would be possible on your laptop.
 
In order to access these you need to log in to your assigned server
 
  
Please download the following programs as needed unless it is installed by default
+
==Downloads==
 +
Please download and install the following programs:
  
 
* '''Windows'''
 
* '''Windows'''
 
** [http://cyberduck.ch/ Cyberduck] (file transfer via sFTP)
 
** [http://cyberduck.ch/ Cyberduck] (file transfer via sFTP)
 
** [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] (SSH client)
 
** [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] (SSH client)
** [http://cygwin.com/install.html Cygwin] (unix interface for windows)
+
** [https://git-for-windows.github.io/ Git for Windows] (This is a UNIX emulator for running command-line programs)
 +
** [https://notepad-plus-plus.org/download/v7.7.1.html Notepad++]
 +
After you have installed Notepad++, open it.
 +
Go to Settings, Preferences, New Document, and set the line endings to Unix.
 +
** [https://web.archive.org/web/20141007105432/http://pbil.univ-lyon1.fr/software/seaview_data/seaview4.exe Seaview]
 +
** [https://github.com/rambaut/figtree/releases figtree] (use the v1.4.4.zip release)
  
 
* '''Mac Os X'''
 
* '''Mac Os X'''
 +
** SCP and SSH installed by default
 
** [http://cyberduck.ch/ Cyberduck] (file transfer via sFTP)
 
** [http://cyberduck.ch/ Cyberduck] (file transfer via sFTP)
** SCP and SSH installed by default
+
** [https://www.barebones.com/products/bbedit/ BBEdit] (BBEdit is the 64 bit version of TextWrangler for contemporary Mac OS)
 +
** [https://web.archive.org/web/20141007105432/http://pbil.univ-lyon1.fr/software/seaview_data/seaview4.zip Seaview]
 +
** [https://github.com/rambaut/figtree/releases figtree] (use the v1.4.4.dmg release)
  
 
* '''Linux'''
 
* '''Linux'''
 
** SCP and SSH installed by default
 
** SCP and SSH installed by default
 +
** [https://web.archive.org/web/20141007105432/http://pbil.univ-lyon1.fr/software/seaview_data/seaview4.tgz Seaview]
 +
** [https://github.com/rambaut/figtree/releases figtree] (use the v1.4.4.tgz release)
 +
** You're set!
  
== Plain-text editors ==
+
Everybody should download Tracer too, which will be used for Bayesian divergence time analyses:
Programs that you might be used to using for manipulating text (e.g., Microsoft Word) do all sorts of complicated things to your files, including inserting lots of weird codes for specific formatting, special software-specific characters, page margin instructions, etc. The software that we are going to use frequently are controlled by text files (that's where their instructions are stored), and they are easily confused--they need a simple file with just their instructions, and none of that other stuff. So we need a plain-text editor to make and manipulate these files.
+
[https://github.com/beast-dev/tracer/releases/tag/v1.7.1 Tracer]
  
Note: different operating systems code line-break or end-of-line characters in different ways, and this can cause problems (Unix: LF '\n', Mac: CR '\r', Windows: CR+LF '\r\n').
+
We will be using more software throughout the workshop than what we're asking you to download here, all of which should be accessible on the remote servers. Feel free to download and use any and all software on your personal device as well, just know that not everything will necessarily finish running in the allotted time, depending on your device.
  
==== With GUI ====
+
== Remote computer access software ==
* '''Linux:''' [http://projects.gnome.org/gedit/ gedit] (GNOME default), [http://kate-editor.org/ Kate/KWrite] (KDE default)
+
We will use Secure Shell (SSH) and sFTP to connect to the servers.
* '''Mac:''' TextEdit (default), [http://www.barebones.com/products/textwrangler/ TextWrangler]
+
The servers are powerful computers where we can run the programs much faster than would be possible on your laptop.
* '''Windows:''' Notepad (default), [http://notepad-plus-plus.org/ Notepad++]
+
In order to access these you need to log in to your assigned server
 
 
==== Without GUI ====
 
[http://www.nano-editor.org/ nano], [http://www.gnu.org/software/emacs/ Emacs], [http://www.vim.org/index.php Vim]
 
  
 
== SSH ==
 
== SSH ==
 
SSH stands for "Secure Shell." These are programs that provide a Unix shell so that one can enter commands and log onto other computers (i.e., those on the server where we will be doing our analyses).
 
SSH stands for "Secure Shell." These are programs that provide a Unix shell so that one can enter commands and log onto other computers (i.e., those on the server where we will be doing our analyses).
=== SSH on Windows ===
+
 
Use PuTTY.
+
'''You must be on the MBL-REGISTERED wifi network or this will not work'''
  
 
=== SSH on Linux and Mac Os X ===
 
=== SSH on Linux and Mac Os X ===
Line 52: Line 79:
 
* '''Linux:''' Konsole (KDE), gnome-terminal (GNOME)
 
* '''Linux:''' Konsole (KDE), gnome-terminal (GNOME)
 
* '''Mac:''' Terminal (in /Applications/Utilities)
 
* '''Mac:''' Terminal (in /Applications/Utilities)
 +
<br>
 +
First we want to log on to the class servers.<br>
 +
In the following command replace <username> and <server name> with the user (first initial plus last name) and server name (class-##) which you can find on the sticker attached to the back of your name tag:<br>
  
In the following command replace username and servername with the user and server name found on the back of your name tag:
 
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
 
ssh username@servername
 
ssh username@servername
 
</pre>
 
</pre>
  
For example if my username is cmeehan and my server name is class-04 I type:<br>
+
For example if my username is jbloggs and my server name is class-04 I type:<br>
ssh cmeehan@class-04
+
ssh jbloggs@class-04.jbpc-np.mbl.edu
 +
It will then ask for your password which is on the back of your card under your username
  
It will then ask for my password which is on the back of your card under your username
+
=== SSH on Windows ===
 +
On windows you can use git for windows or PuTTY for accessing the class servers.<br>
 +
If using git for windows, follow the Linux/Mac approach above. <br>
 +
To set up SSH on PuTTY
 +
Enter your username and servername in the "Host Name or IP Address" field
 +
You can find your username and servername on the piece of paper:
 +
<pre style="font-size: 1.25em">
 +
username@servername
 +
</pre>
  
==== Changing your password ====
+
For example if my username is jbloggs and my server name is class-02 I type:<br>
The first thing to do once you have successfully logged on to a server is to change your password. This is done by typing:<br>
 
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
passwd
+
jbloggs@class-02.jbpc-np.mbl.edu
 
</pre>
 
</pre>
This will prompt you to enter a new password so do so and press the enter key. Next you re-enter the new password and again press enter. It may then ask you for your LDAP password and you should type in your original password given on the back of your card.<br>
 
Once you have done this every time you ssh in to the server you will use the new password created.
 
  
== Directory structure ==
+
Enter a name for this configuration (e.g. MolEvol) where it says "Saved Sessions" and click Save
The file systems used by Linux, Mac OS X, and Windows are organized in a hierarchical, multifurcating tree structure. That might sound confusing, but you're used to working with this organization scheme through the Mac Finder or the Windows Explorer--folders (directories) are stored inside other folders, and they in yet other folders. The path through this directory tree can be used to specify the absolute (starting at the root) or relative (to some other directory) location of any given file.
+
Now you can access the cluster by clicking on that saved session.
 +
 
 +
PuTTy copy paste tip: [http://www.chiark.greenend.org.uk/~sgtatham/putty/faq.html#faq-cutpaste PuTTy FAQ]
  
*NOTE: Regardless of the operating system on your laptop, when you log on to the cluster, you will be on the class machines, and they're all running linux.
+
Copy and paste works similarly to the X Window System. You use the left mouse button to select text in the PuTTY window. The act of selection automatically copies the text to the clipboard: there is no need to press Ctrl-Ins or Ctrl-C or anything else. In fact, pressing Ctrl-C will send a Ctrl-C character to the other end of your connection (just like it does the rest of the time), which may have unpleasant effects. The only thing you need to do, to copy text to the clipboard, is to select it.
  
==== Linux/Mac ====
+
To paste the clipboard contents into a PuTTY window, by default you click the right mouse button. If you have a three-button mouse and are used to X applications, you can configure pasting to be done by the middle button instead, but this is not the default because most Windows users don't have a middle button at all.
The root folder of the directory tree is symbolized by a forward slash "/". Path names through the directory tree are formed by separating directory names with the "/" character. For example in the path '''/usr/local''', '''local''' is a ''subdirectory'' of '''usr''', which in turn is a subdirectory of the root folder ('''/'''). Further, '''usr''' is called the ''parent'' directory of '''local''' (whose parent is '''/'''). Users with an account on the computer have a so called ''home'' directory for their own files and folders, and where the operating system keeps user-specific settings. The operating system uses other locations in the directory tree to store system-wide files, like some executables, system-wide settings, etc.
 
  
==== Windows ====
+
You can also paste by pressing Shift-Ins.
In Windows, each disk ''partition'' has its own, separate root (without a common root for all partitions). However, each drive still has a most elemental directory, e.g. denoted '''C:\''' for the partition named C. Absolute paths are therefore based on the partition root. Folder names in paths are separated by the backslash character "\".
 
  
==== Current working directory ====
+
=== Everyone ===
The current working directory (or working directory) may be defined as the directory we are working in at the moment. Path names that do not start with '''/''' are interpreted as relative to the current working directory.
+
You should have a black screen asking for your password, which is on the piece of paper. Type it in. You will not see the letters you are typing.
  
== Command line interface ==
+
==== Changing your password ====
When you open up your SSH client (a Terminal window on a Mac, or PuTTY for Windows), you'll see a prompt that will look vaguely like: Macintosh-6:user$ There are lots of variations on the theme, but the prompt usually has a little bit of information on where you currently are on the computer (in this case, in a folder called "user"), and then some sort of symbol, and then a space where you can enter commands. Commands are only executed after you press enter. If you are logged onto a class server, the prompt will be something like [username@classServerName ~]$
+
The first thing to do once you have successfully logged on to a server is to change your password. This is done by typing:<br>
 +
<pre style="font-size: 1.25em">
 +
passwd
 +
</pre>
 +
This will prompt you to enter a new password so do so and press the enter key. Next you re-enter the new password and again press enter. It may then ask you for your LDAP password and you should type in your original password given on the back of your card.<br>
 +
Once you have done this every time you ssh in to the server you will use the new password created.
  
 
====Basic Syntax====
 
====Basic Syntax====
 
Unix commands follow the general format of:  
 
Unix commands follow the general format of:  
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
command -options target.
+
command -options target
 
</pre>
 
</pre>
 
Not all commands need options (sometimes called flags, and generally preceded by a single or double hyphen ("-" or "--")) or targets, but others require them.
 
Not all commands need options (sometimes called flags, and generally preceded by a single or double hyphen ("-" or "--")) or targets, but others require them.
Line 104: Line 144:
 
*The tilde (~) refers to your home directory. On the class machines your home directory is /class/yourusername. You'll also have a unique home directory on your laptop, etc. The tilde is very helpful if you get lost while using the terminal -- just type "cd ~" and you'll be back in your homedirectory.
 
*The tilde (~) refers to your home directory. On the class machines your home directory is /class/yourusername. You'll also have a unique home directory on your laptop, etc. The tilde is very helpful if you get lost while using the terminal -- just type "cd ~" and you'll be back in your homedirectory.
 
*A forward slash (/) by itself or at the start of a path refers to the root of the filing system -- the folder that contains all other folders.
 
*A forward slash (/) by itself or at the start of a path refers to the root of the filing system -- the folder that contains all other folders.
*NOTE: do not make changes on the class server in the root folder or any shared folders. All your work is to be done in your home directory or a subdirectory of this.
+
*NOTE: do not make changes on the class server in the root folder or any shared folders. All your work is to be done in your home directory or a subdirectory.
  
====Some suggestions concerning file and folder names====
+
==Intro-to-Unix tutorial==
*Avoid spaces in script and filenames (use underscores, dots, or hyphens, use "CamelBack" notation). Spaces are used in command line editing to separate options etc so if there is a space in a filename it will mess up the correct running of programs.
 
*Do not use "weird" characters (#@!*&^, etc., especially ?, *, \, or /)
 
 
 
====Don't Panic====
 
When it all goes south, "control-C" is your friend. It breaks whatever processes are running, and gives you your prompt back. Or, failing that, just close the Terminal and start again.
 
  
==Intro-to-Unix tutorial==
+
Please go to the [[UNIX tutorial]] and go through the step by step guide there.
  
====Navigating====
+
==Downloading files==
Start by entering
+
In order to get a file from an online source you can use wget. Type:
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
pwd
+
wget <URL>
 
</pre>
 
</pre>
This will print your working directory (the directory you are currently in). You should be in /class/your_login<br>
+
Where the url is the website address of the file you wish to download. For example
Next type
 
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
ls
+
wget https://raw.githubusercontent.com/gtiley/misc/master/mbl2019.txt
 
</pre>
 
</pre>
This list the contents of your working directory (which is likely empty).<br>
+
Will download the file mbl2019.txt to the directory you are in.
You can also look at the contents of any other directory by supplying the path. For instance,
+
Note: If you are on OSX and trying to do this locally to your machine, this won't work. Curl is an alternative for doing this. See if you can figure out using google how curl works. Wget will still work on the class servers, regardless of your operating system.
<pre style="font-size: 1.25em">
+
 
ls ..
+
Copy this file to a file named <yourname.txt> (remember! No spaces!)
</pre>
+
 
will list the contents of the parent directory that your current directory is in, in this case the class directory.
+
Edit the statement in the file using nano, to a fact about yourself.
 +
 
 +
Copy it to the /class/molevol-shared/facts/2019 directory.
 +
 
 +
'''Ask a person next to you their name, and find and read their fact!'''
 +
 
 +
==Setting up Cyberduck==
 +
====Creating a bookmark====
 +
When you open Cyberduck a screen should appear that has some default connections bookmarked such as Google Docs and Amazon.<br />
 +
Click the ''' '+' ''' button at the ''' BOTTOM LEFT ''' to create a new bookmark (NOT the one with the globe at the top). Here we will put the details of our cluster login and save it so that each time we open Cyberduck we don't have to re-enter the details.
 +
* Click the dropdown menu that says FTP (file Transfer Protocol) and select SFTP. <br />
 +
 
 +
* Create a nickname for this bookmark. For example mine is called 'MBL cluster'.<br />
 +
 
 +
* In 'server' write the class number of the server you were given. For example, I am rwilliams@class-04 so I put class-04 in here. <br />
 +
 
 +
* In the username box place the username you were given. This is likely your first initial and your last name. For example mine is rwilliams. <br />
 +
 
 +
Once all these details are entered close that screen. (e.g. on a mac click the red circle or on windows click the X).<br />
 +
A new bookmark should have appeared named whatever you set as the nickname. Double click on this.<br />
 +
Cyberduck should ask you for the password. Type in the password you changed it to earlier. There should be a button asking for you to 'add to your keychain' or something similar. Make sure this box is ticked. Once done, click to connect.<br />
 +
This may connect you directly to the cluster. If not double click on the bookmark again.<br />
 +
You now should see your home directory on the cluster. You should see the file "yourname.txt" that we created earlier.
 +
 
 +
===Navigating the cluster on Cyberduck and transferring files===
 +
Navigating through the class cluster in cyberduck is easy. <br />
 +
Double click on a folder to open that folder.<br />
 +
Above the window that lists files there should be your path. For example when I login my path shows '/classfs/rwilliams'. This tells me where I am on the cluster right now. If you click this dropdown menu you can see each folder you are in. For example when I click on the dropdown I have a line that says /classfs/rwilliams, one that says /class and one that just says /. Clicking on any of these will allow you to navigate to that folder. If you get lost in folders you can click this dropdown and select /class/username (where username is your own username) and this will bring you to your home directory.<br />
 +
An alternative way to go to a specific folder is click the ''' 'Go' ''' menu at the top of Cyberduck and click ''' 'Go to folder' '''. Here you can write the full path and go to a specific folder. For example, if you are logged in to the cluster through a terminal/putty you can type 'pwd' and then copy that path into the 'go to folder' box and now Cyberduck will be in the same folder as your terminal window.
 +
 
 +
There are two ways to download files from the class cluster to your computer.<br />
 +
'''1.''' If you double click on a file name, this file will be automatically downloaded to your default download folder. A transfer window should open up when you double click on a file. At the bottom there will be a line that starts 'local file'. This is the location of that downloaded file on your own computer<br />
 +
'''2.''' An easier way to get files both to and from the class cluster through Cyberduck is through drag and drop. If you have an open finder window (mac) or explorer window (windows) you can click and drag a file from the Cyberduck window to the finder/explorer window directly. You also can do this in the reverse (drag a file from finder/explorer to Cyberduck) to place a file on the cluster in that folder.
 +
 
 +
===Using Cyberduck for remote editing===
 +
Instead of using nano or similar to edit files on the class cluster you can use a GUI text editor like TextWrangler through Cyberduck.<br />
 +
In cyberduck click ''' 'Cyberduck' '''->''' 'Preferences' ''' on a Mac or ''' 'Edit' '''-> ''' 'Preferences' ''' on Windows.<br />
 +
Click the ''' 'Editor'''' button.<br />
 +
There should be a dropdown menu where you can select a text editor. Select TextWrangler on a Mac, Notepad++ on Windows.<br />
 +
Close this window.<br />
 +
Now when you single click on a file in Cyberduck, a 'Edit' button with the icon for the editor you selected should appear in the top right corner. Click this button to open that file in the editor. Now you can edit the file on the cluster without downloading to your local computer. When you save this file it will be automatically updated on the class cluster.
 +
 
 +
==Short Cyber Duck Exercise==
  
mkdir is the command to make a directory. Type
+
The easiest way to transfer files using CyberDuck is with drag-and-drop. Try to drag a file from your desktop (or another folder of your choosing) from the CyberDuck window. This method allows you to choose unique destinations for each file you transfer.
<pre style="font-size: 1.25em">
 
mkdir myfolder
 
</pre>
 
to make a new folder called myfolder. Type ls and then enter. It should be listed. We can also use the ls command with flags at this point. Typing
 
<pre style="font-size: 1.25em">
 
ls -l
 
</pre>
 
will list the contents of the current directory in "long" format which includes information about permissions and file size.
 
  
cd is the command to change directories. We can move into the new folder you made by typing
+
In CyberDuck, double click example.txt your 'myfolder' on the cluster. To where does this file download? CyberDuck also allows for more permanent modification to how files are downloaded, you can set automatic download locations. If you want to change this, under Edit -> Preferences -> Transfers, you can edit this.
<pre style="font-size: 1.25em">
 
cd myfolder
 
</pre>
 
You can use pwd to confirm you've moved and are now in a new working directory. You can move back to your home directory by typing
 
<pre style="font-size: 1.25em">
 
cd ..
 
</pre>
 
And can move to the root class directory by typing
 
<pre style="font-size: 1.25em">
 
cd /
 
</pre>
 
Confirm you are in the class directory with pwd. You can move from here to the myfolder directory in your home directory by typing
 
<pre style="font-size: 1.25em">
 
cd class/cmeehan/myfolder
 
</pre>
 
  
====Calling a program====
+
Lastly: Right clicking (Ctrl click on a Mac) will allow you to select Download to or Download As. This will allow you to choose a location, and if you choose, a new file name for the file.
First, let's make a file using the command-line editor "nano." If you use a pre-existing file as nano's target, it will open it for editing, and if you use a non-existent file name, nano will create it for you (a new blank file). Let's do the latter:
 
<pre style="font-size: 1.25em">
 
nano firstprogram.sh
 
</pre>
 
This command both opens nano, and creates a file called "firstprogram.sh". In the nano window, enter the following two lines:
 
<pre style="font-size: 1.25em">
 
#!/bin/bash
 
echo "hello mbl"
 
</pre>
 
This file is called a shell script. It is a file that contains lists of unix commands like echo, ls, etc.<br>
 
To move around your nano window, use the arrow keys, not your mouse. Control+x will exit, and nano will ask you if you want to save changes. Say yes. <br>
 
  
Shell scripts almost always have the suffix ".sh". They are run by typing "sh" before the filename.<br>
+
These methods will all work to get files from the cluster to your computer, and each have their own benefits. Generally, you will be fine using any of them throughout the course.
We can run this program by typing
 
<pre style="font-size: 1.25em">
 
sh firstprogram.sh
 
</pre>
 
  
This should print 'hello mbl' to the screen.
+
Make a folder on your computer to hold all of your lab materials from this course!
  
====Copying, renaming, and moving files====
+
==SCP is a good alternative to Cyberduck for linux and Mac's==
The copy command (cp) is used to copy files to new places. The command basic syntax is '''cp source_file destination_file'''
+
There are two places you may need to get files on to the class server from: your own computer or an online source<br>
First create a file called 'tmp1.txt' in nano and put whatever you want inside of it.<br>
+
In order to get files from your computer to the server open a terminal window and navigate to the folder on your computer using the commands like cd. Once in the folder containing the file you want to upload you type
We will now make a copy of tmp1.txt called tmp2.txt by typing:
 
<pre style="font-size: 1.25em">
 
cp tmp1.txt tmp2.txt
 
</pre>
 
We can also cp a file from the shared directory using absolute and relative paths.
 
<pre style="font-size: 1.25em">
 
cp /class/shared/testfile.txt .
 
</pre>
 
will copy a file named "testfile.txt" to your current directory but will not change its name. Use ls to verify.<br>
 
The move command (mv) can be used to move or rename files. The command syntax is '''mv source destination'''
 
<pre style="font-size: 1.25em">
 
mv testfile.txt example.txt
 
</pre>
 
has the effect of renaming testfile.txt to example.txt. Use ls to check.
 
We can move this file up one directory by using mv as well.
 
 
<pre style="font-size: 1.25em">
 
<pre style="font-size: 1.25em">
mv example.txt ..
+
scp filename username@classServername.jbpc-np.mbl.edu:~
 
</pre>
 
</pre>
will move the example.txt file to the parent directory of your current directory. Use ls .. to check.
+
e.g.
*NOTE: Do not move files that are not in your home directory or a subdirectory of this. All files in shared or root folders are to be copied, never moved.
+
just like cp, the format is scp <from> <to>
  
 +
To get files from your computer to the cluster:
  
====Loading pre-installed programs====
+
    scp text.txt gtiley@class-04.jbpc-np.mbl.edu:/class/gtiley
Many of the programs that you will need for the course are already installed on the class servers. These need to be specifically loaded in for them to work.<br>
 
For example on the command line type
 
<pre style="font-size: 1.25em">
 
blastn
 
</pre>
 
You should get an error message saying: '''-bash: blastn: command not found'''<br>
 
Now type
 
<pre style="font-size: 1.25em">
 
module load bioware
 
</pre>
 
This should load all the preinstalled programs so that you can access them. Now type blastn again and you should see:<br>
 
'''BLAST query/options error: Either a BLAST database or subject sequence(s) must be specified'''<br>
 
Showing that blastn is now available for use.<br>
 
If needed, programs can all be unloaded by typing '''module unload bioware'''<br>
 
  
 +
To get files from the cluster to your computer:
  
 +
    scp gtiley@class-04.jbpc-np.mbl.edu:/class/gtiley/testfile.txt .
  
====Useful links====
+
==Useful links==
 
The following table contains a list of commands that will allow us to navigate through the directory structure. The entries are linked to their [http://www.wikipedia.org/ Wikipedia] pages, which contain very useful examples.
 
The following table contains a list of commands that will allow us to navigate through the directory structure. The entries are linked to their [http://www.wikipedia.org/ Wikipedia] pages, which contain very useful examples.
  
Line 230: Line 242:
 
|-
 
|-
 
! style="background:#AACCEE;" | Linux/Mac
 
! style="background:#AACCEE;" | Linux/Mac
! style="background:#AACCEE;" | MS-DOS
 
 
! style="background:#AACCEE;" class="unsortable" | Description
 
! style="background:#AACCEE;" class="unsortable" | Description
 
! style="background:#AACCEE;" class="unsortable" | Syntax (Linux/Mac)
 
! style="background:#AACCEE;" class="unsortable" | Syntax (Linux/Mac)
 
|-
 
|-
 
| [http://en.wikipedia.org/wiki/Pwd pwd]
 
| [http://en.wikipedia.org/wiki/Pwd pwd]
| [http://en.wikipedia.org/wiki/Cd_(command) chdir]
 
 
| print working directory
 
| print working directory
 
| '''pwd'''
 
| '''pwd'''
 
|-
 
|-
 
| [http://en.wikipedia.org/wiki/Ls ls]
 
| [http://en.wikipedia.org/wiki/Ls ls]
| [http://en.wikipedia.org/wiki/Dir_(command) dir]
 
 
| list directory contents
 
| list directory contents
 
| '''ls'''
 
| '''ls'''
 
|-
 
|-
| [http://en.wikipedia.org/wiki/History(Unix) history]
+
| [https://en.wikipedia.org/wiki/History_(command) history]
| [http://en.wikipedia.org/wiki/DOSKey doskey /history]
 
 
| display command history
 
| display command history
 
| '''history'''
 
| '''history'''
 
|-
 
|-
  
| [http://en.wikipedia.org/wiki/Cd_(command) cd]
 
 
| [http://en.wikipedia.org/wiki/Cd_(command) cd]
 
| [http://en.wikipedia.org/wiki/Cd_(command) cd]
 
| change directory
 
| change directory
 
| '''cd''' ''directory_name''
 
| '''cd''' ''directory_name''
 
|-
 
|-
| [http://en.wikipedia.org/wiki/Mkdir mkdir]
 
 
| [http://en.wikipedia.org/wiki/Mkdir mkdir]
 
| [http://en.wikipedia.org/wiki/Mkdir mkdir]
 
| make directory
 
| make directory
Line 261: Line 267:
 
|-
 
|-
 
| [http://en.wikipedia.org/wiki/Cp_(Unix) cp]
 
| [http://en.wikipedia.org/wiki/Cp_(Unix) cp]
| [http://en.wikipedia.org/wiki/Copy_(command) copy]
 
 
| copy files
 
| copy files
 
| '''cp''' ''original_filename copied_filename''
 
| '''cp''' ''original_filename copied_filename''
 
|-
 
|-
 
| [http://en.wikipedia.org/wiki/Mv mv]
 
| [http://en.wikipedia.org/wiki/Mv mv]
| [http://en.wikipedia.org/wiki/Move_(command) move]
 
 
| move files (the same as rename files)
 
| move files (the same as rename files)
 
| '''mv''' ''original_filename moved_filename''
 
| '''mv''' ''original_filename moved_filename''
|-
 
| [http://en.wikipedia.org/wiki/Rm_(Unix) rm]
 
| [http://en.wikipedia.org/wiki/Del_(command) del]
 
| remove file(s)
 
| '''rm''' ''filename''
 
 
|-
 
|-
 
| [http://en.wikipedia.org/wiki/Clear_(Unix) clear]
 
| [http://en.wikipedia.org/wiki/Clear_(Unix) clear]
| [http://en.wikipedia.org/wiki/Cls_(computing) cls]
 
 
| clear the screen
 
| clear the screen
 
| '''clear'''
 
| '''clear'''
 
|-
 
|-
| [http://en.wikipedia.org/wiki/Exit_(command) exit]
 
 
| [http://en.wikipedia.org/wiki/Exit_(command) exit]
 
| [http://en.wikipedia.org/wiki/Exit_(command) exit]
 
| quit command line
 
| quit command line
 
| '''exit'''
 
| '''exit'''
 +
|-
 +
| ' '
 +
| Ctrl-E
 +
| '''Go to the end of line'''
 +
|-
 +
|' '
 +
| Ctrl-A
 +
| '''Go to the beginning of line'''
 
|}
 
|}
  
 
* UNIX/Linux command line cheat-sheet: [http://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/ http://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/]
 
* UNIX/Linux command line cheat-sheet: [http://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/ http://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/]
 
* MS-DOS full command list: [http://en.wikipedia.org/wiki/List_of_MS-DOS_commands MS-DOS commands]
 
* MS-DOS full command list: [http://en.wikipedia.org/wiki/List_of_MS-DOS_commands MS-DOS commands]
 
+
* A unix shell tutorial from Software Carpentry: [http://swcarpentry.github.io/shell-novice/]
==== Command line editing ====
+
<!-- * More information from last year [https://molevol.mbl.edu/wiki/index.php/Computer_lab_introduction] -->
The following features of most command line interpreters often come in handy:
 
 
 
* Up and down arrow keys: cycle through previously issued commands
 
* [http://en.wikipedia.org/wiki/Tab_completion Tab completion]
 
* 'CTRL+a' moves cursor to beginning of the line
 
* 'CTRL+e' moves cursor to end of the line
 
 
 
== Advanced topics (unfinished) ==
 
 
 
* [http://cygwin.com/ Cygwin] (a Linux look-and-feel environment for Windows)
 
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PSCP] (SCP client for Windows)
 
 
 
=== Adding a directory to the path ===
 
 
 
==== sed, grep, and awk ====
 
* sed (UNIX stream editor)
 
** [http://sed.sourceforge.net/sed1line.txt sed oneliners]
 
 
 
=== Regular expressions ===
 
* Regular expressions cheat-sheet: [http://www.addedbytes.com/cheat-sheets/regular-expressions-cheat-sheet/ http://www.addedbytes.com/cheat-sheets/regular-expressions-cheat-sheet/]
 
 
 
== More about text editors ==
 
 
 
====Looking inside text files====
 
There are different commands that can be used to look inside files from within the command line interface. Two of these are [http://en.wikipedia.org/wiki/Cat_(Unix) cat] in Linux/Mac, [http://en.wikipedia.org/wiki/Type_(command) type] in MS-DOS, and [http://en.wikipedia.org/wiki/Less_(Unix) less] in Linux/Mac, [http://en.wikipedia.org/wiki/More_(command) more] in MS-DOS. The command [http://en.wikipedia.org/wiki/More_(command) more] is also available in Linux/Mac, and in fact [http://en.wikipedia.org/wiki/More_(command) more] is [http://en.wikipedia.org/wiki/Less_(Unix) less], but [http://en.wikipedia.org/wiki/Less_(Unix) less] has more features than [http://en.wikipedia.org/wiki/More_(command) more].
 
*'''cat''': ''cat'' stands for concatenate.  This command is useful for peeking at short files; using ''cat'' to view a long file results in the top lines scrolling off before one can even read them. The simplest use of ''cat'' is with <pre style="font-size: 1.25em">cat filename</pre>
 
*'''less''': This command is useful to read long text files because a page of text (i.e., a command line window filled from top to bottom) is displayed one at a time. With ''less'' is easy to move forward and backward by lines, pages and even between two or more files.  ''less'' is a program in itself, so when it is invoked, a prompt appears at the bottom of the page awaiting for a new ''less'' command.  The prompt in ''less'' is a colon (''':''').  The simplest use of ''less'' is with <pre style="font-size: 1.25em">less filename</pre>
 
The following table contains a list of useful ''less'' commands
 
 
 
{| class="wikitable sortable"
 
|+Some basic ''less'' commands
 
|-
 
! style="background:#AACCEE;" | Command
 
! style="background:#AACCEE;" class="unsortable" | Description
 
|-
 
| spacebar
 
| display next page
 
|-
 
| return
 
| display next line
 
|-
 
| ''n'' f
 
| move forward ''n'' lines
 
|-
 
| b
 
| move backward one page
 
|-
 
| ''n'' b
 
| move backward ''n'' lines
 
|-
 
| / ''word''
 
| search forward for ''word''
 
|-
 
|? ''word''
 
| search backward for ''word''
 
|-
 
| h
 
| help
 
|-
 
| q
 
| quit
 
|}
 
==Online resources to learn UNIX==
 
[http://www.ee.surrey.ac.uk/Teaching/Unix/ Unix for beginners]
 
 
 
[http://snap.nlc.dcccd.edu/learn/idaho/unixindex.html A basic Unix tutorial]
 
 
 
[http://www.doc.ic.ac.uk/~wjk/UnixIntro/ Introduction to Unix]
 

Latest revision as of 19:02, 2 August 2019

"On the other side of the screen, it all looks so easy." (TRON, 1982)

Goals

By the end of this introduction, you should be able to:

  • Log in to the class cluster.
  • Navigate and understand the directory structure.
  • Create and edit files and directories on the cluster.
  • Move files to the cluster.

Ask for help if any of these things aren't working by the end!

Slack group

Slack is a messaging/discussion program that allows for large groups to interact as needed.
There is a slack group set up for people to ask questions and to have discussions.
The slack team is MBL-MOLE-2019
You can use the Slack app on your laptop and your phone.
If you can't join through the app/website, you can click on this link (note that until Sunday, 28 July 2019, this link was for the 2018 version of the course, but it is now fixed).

WiFi Login

To connect to the cluster you must:

1. be logged on to the MBL-REGISTER wifi network.

2. have any virtual private network (VPN) turned off.

Full instructions can be found here

When you open a new browser window after you've registered, you should see a prompt for name and password. Your username is your initials followed by the 5 digit number on the side of your identification card. Your password is the same. E.g. if your name is Jane Bloggs and your card has the number 12345 on the side then your login details are:

username: jb12345
password: jb12345

Some people have been having trouble, if it isn't working find or email George (george.tiley@gmail.com) or any TA, or ask on the Slack group.

Downloads

Please download and install the following programs:

After you have installed Notepad++, open it. Go to Settings, Preferences, New Document, and set the line endings to Unix.

  • Mac Os X
    • SCP and SSH installed by default
    • Cyberduck (file transfer via sFTP)
    • BBEdit (BBEdit is the 64 bit version of TextWrangler for contemporary Mac OS)
    • Seaview
    • figtree (use the v1.4.4.dmg release)
  • Linux
    • SCP and SSH installed by default
    • Seaview
    • figtree (use the v1.4.4.tgz release)
    • You're set!

Everybody should download Tracer too, which will be used for Bayesian divergence time analyses: Tracer

We will be using more software throughout the workshop than what we're asking you to download here, all of which should be accessible on the remote servers. Feel free to download and use any and all software on your personal device as well, just know that not everything will necessarily finish running in the allotted time, depending on your device.

Remote computer access software

We will use Secure Shell (SSH) and sFTP to connect to the servers. The servers are powerful computers where we can run the programs much faster than would be possible on your laptop. In order to access these you need to log in to your assigned server

SSH

SSH stands for "Secure Shell." These are programs that provide a Unix shell so that one can enter commands and log onto other computers (i.e., those on the server where we will be doing our analyses).

You must be on the MBL-REGISTERED wifi network or this will not work

SSH on Linux and Mac Os X

First, open a terminal window:

  • Linux: Konsole (KDE), gnome-terminal (GNOME)
  • Mac: Terminal (in /Applications/Utilities)


First we want to log on to the class servers.
In the following command replace <username> and <server name> with the user (first initial plus last name) and server name (class-##) which you can find on the sticker attached to the back of your name tag:

ssh username@servername

For example if my username is jbloggs and my server name is class-04 I type:

ssh jbloggs@class-04.jbpc-np.mbl.edu

It will then ask for your password which is on the back of your card under your username

SSH on Windows

On windows you can use git for windows or PuTTY for accessing the class servers.
If using git for windows, follow the Linux/Mac approach above.
To set up SSH on PuTTY Enter your username and servername in the "Host Name or IP Address" field You can find your username and servername on the piece of paper:

username@servername

For example if my username is jbloggs and my server name is class-02 I type:

jbloggs@class-02.jbpc-np.mbl.edu

Enter a name for this configuration (e.g. MolEvol) where it says "Saved Sessions" and click Save Now you can access the cluster by clicking on that saved session.

PuTTy copy paste tip: PuTTy FAQ

Copy and paste works similarly to the X Window System. You use the left mouse button to select text in the PuTTY window. The act of selection automatically copies the text to the clipboard: there is no need to press Ctrl-Ins or Ctrl-C or anything else. In fact, pressing Ctrl-C will send a Ctrl-C character to the other end of your connection (just like it does the rest of the time), which may have unpleasant effects. The only thing you need to do, to copy text to the clipboard, is to select it.

To paste the clipboard contents into a PuTTY window, by default you click the right mouse button. If you have a three-button mouse and are used to X applications, you can configure pasting to be done by the middle button instead, but this is not the default because most Windows users don't have a middle button at all.

You can also paste by pressing Shift-Ins.

Everyone

You should have a black screen asking for your password, which is on the piece of paper. Type it in. You will not see the letters you are typing.

Changing your password

The first thing to do once you have successfully logged on to a server is to change your password. This is done by typing:

passwd

This will prompt you to enter a new password so do so and press the enter key. Next you re-enter the new password and again press enter. It may then ask you for your LDAP password and you should type in your original password given on the back of your card.
Once you have done this every time you ssh in to the server you will use the new password created.

Basic Syntax

Unix commands follow the general format of:

command -options target

Not all commands need options (sometimes called flags, and generally preceded by a single or double hyphen ("-" or "--")) or targets, but others require them.

  • For example:
    • cd homedirectory uses the command "cd" (change directory) and the target "homedirectory" to move from the current directory into the subdirectory called "homedirectory"
    • ls -l homedirectory uses the command "ls" (list), the option "-l" for long-list, and the target "homedirectory" to list the contents of homedirectory in the "long list" format, which provides more thorough descriptions than does the regular "ls".

Notes on syntax for directory structure

  • Two dots (..) indicates the parent directory of the present working directory. So, for example, "cd .. will move you back one directory.
  • One dot (.) indicates the present working directory. So, for example, "cd ." will keep you where you are. There are times where the single dot can be more useful than this...
  • The tilde (~) refers to your home directory. On the class machines your home directory is /class/yourusername. You'll also have a unique home directory on your laptop, etc. The tilde is very helpful if you get lost while using the terminal -- just type "cd ~" and you'll be back in your homedirectory.
  • A forward slash (/) by itself or at the start of a path refers to the root of the filing system -- the folder that contains all other folders.
  • NOTE: do not make changes on the class server in the root folder or any shared folders. All your work is to be done in your home directory or a subdirectory.

Intro-to-Unix tutorial

Please go to the UNIX tutorial and go through the step by step guide there.

Downloading files

In order to get a file from an online source you can use wget. Type:

wget <URL>

Where the url is the website address of the file you wish to download. For example

wget https://raw.githubusercontent.com/gtiley/misc/master/mbl2019.txt

Will download the file mbl2019.txt to the directory you are in. Note: If you are on OSX and trying to do this locally to your machine, this won't work. Curl is an alternative for doing this. See if you can figure out using google how curl works. Wget will still work on the class servers, regardless of your operating system.

Copy this file to a file named <yourname.txt> (remember! No spaces!)

Edit the statement in the file using nano, to a fact about yourself.

Copy it to the /class/molevol-shared/facts/2019 directory.

Ask a person next to you their name, and find and read their fact!

Setting up Cyberduck

Creating a bookmark

When you open Cyberduck a screen should appear that has some default connections bookmarked such as Google Docs and Amazon.
Click the '+' button at the BOTTOM LEFT to create a new bookmark (NOT the one with the globe at the top). Here we will put the details of our cluster login and save it so that each time we open Cyberduck we don't have to re-enter the details.

  • Click the dropdown menu that says FTP (file Transfer Protocol) and select SFTP.
  • Create a nickname for this bookmark. For example mine is called 'MBL cluster'.
  • In 'server' write the class number of the server you were given. For example, I am rwilliams@class-04 so I put class-04 in here.
  • In the username box place the username you were given. This is likely your first initial and your last name. For example mine is rwilliams.

Once all these details are entered close that screen. (e.g. on a mac click the red circle or on windows click the X).
A new bookmark should have appeared named whatever you set as the nickname. Double click on this.
Cyberduck should ask you for the password. Type in the password you changed it to earlier. There should be a button asking for you to 'add to your keychain' or something similar. Make sure this box is ticked. Once done, click to connect.
This may connect you directly to the cluster. If not double click on the bookmark again.
You now should see your home directory on the cluster. You should see the file "yourname.txt" that we created earlier.

Navigating the cluster on Cyberduck and transferring files

Navigating through the class cluster in cyberduck is easy.
Double click on a folder to open that folder.
Above the window that lists files there should be your path. For example when I login my path shows '/classfs/rwilliams'. This tells me where I am on the cluster right now. If you click this dropdown menu you can see each folder you are in. For example when I click on the dropdown I have a line that says /classfs/rwilliams, one that says /class and one that just says /. Clicking on any of these will allow you to navigate to that folder. If you get lost in folders you can click this dropdown and select /class/username (where username is your own username) and this will bring you to your home directory.
An alternative way to go to a specific folder is click the 'Go' menu at the top of Cyberduck and click 'Go to folder' . Here you can write the full path and go to a specific folder. For example, if you are logged in to the cluster through a terminal/putty you can type 'pwd' and then copy that path into the 'go to folder' box and now Cyberduck will be in the same folder as your terminal window.

There are two ways to download files from the class cluster to your computer.
1. If you double click on a file name, this file will be automatically downloaded to your default download folder. A transfer window should open up when you double click on a file. At the bottom there will be a line that starts 'local file'. This is the location of that downloaded file on your own computer
2. An easier way to get files both to and from the class cluster through Cyberduck is through drag and drop. If you have an open finder window (mac) or explorer window (windows) you can click and drag a file from the Cyberduck window to the finder/explorer window directly. You also can do this in the reverse (drag a file from finder/explorer to Cyberduck) to place a file on the cluster in that folder.

Using Cyberduck for remote editing

Instead of using nano or similar to edit files on the class cluster you can use a GUI text editor like TextWrangler through Cyberduck.
In cyberduck click 'Cyberduck' -> 'Preferences' on a Mac or 'Edit' -> 'Preferences' on Windows.
Click the 'Editor' button.
There should be a dropdown menu where you can select a text editor. Select TextWrangler on a Mac, Notepad++ on Windows.
Close this window.
Now when you single click on a file in Cyberduck, a 'Edit' button with the icon for the editor you selected should appear in the top right corner. Click this button to open that file in the editor. Now you can edit the file on the cluster without downloading to your local computer. When you save this file it will be automatically updated on the class cluster.

Short Cyber Duck Exercise

The easiest way to transfer files using CyberDuck is with drag-and-drop. Try to drag a file from your desktop (or another folder of your choosing) from the CyberDuck window. This method allows you to choose unique destinations for each file you transfer.

In CyberDuck, double click example.txt your 'myfolder' on the cluster. To where does this file download? CyberDuck also allows for more permanent modification to how files are downloaded, you can set automatic download locations. If you want to change this, under Edit -> Preferences -> Transfers, you can edit this.

Lastly: Right clicking (Ctrl click on a Mac) will allow you to select Download to or Download As. This will allow you to choose a location, and if you choose, a new file name for the file.

These methods will all work to get files from the cluster to your computer, and each have their own benefits. Generally, you will be fine using any of them throughout the course.

Make a folder on your computer to hold all of your lab materials from this course!

SCP is a good alternative to Cyberduck for linux and Mac's

There are two places you may need to get files on to the class server from: your own computer or an online source
In order to get files from your computer to the server open a terminal window and navigate to the folder on your computer using the commands like cd. Once in the folder containing the file you want to upload you type

scp filename username@classServername.jbpc-np.mbl.edu:~

e.g. just like cp, the format is scp <from> <to>

To get files from your computer to the cluster:

   scp text.txt gtiley@class-04.jbpc-np.mbl.edu:/class/gtiley

To get files from the cluster to your computer:

   scp gtiley@class-04.jbpc-np.mbl.edu:/class/gtiley/testfile.txt .

Useful links

The following table contains a list of commands that will allow us to navigate through the directory structure. The entries are linked to their Wikipedia pages, which contain very useful examples.

Some basic commands
Linux/Mac Description Syntax (Linux/Mac)
pwd print working directory pwd
ls list directory contents ls
history display command history history
cd change directory cd directory_name
mkdir make directory mkdir directory_name
cp copy files cp original_filename copied_filename
mv move files (the same as rename files) mv original_filename moved_filename
clear clear the screen clear
exit quit command line exit
' ' Ctrl-E Go to the end of line
' ' Ctrl-A Go to the beginning of line