Page 1 of 9
High Performance Compute (HPC) Cluster
Copying Data to and from SCIAMA ( pro: Shama )
Author:
G.Burton
Version:
Version 1.1
Date:
Mar 11
http://www.sciama.icg.port.ac.uk
It should be noted that DATA WILL NOT BE BACKED UP ON SCIAMA .
Page 2 of 9
INTRODUCTION
From a user perspective there are two data areas on Sciama. Your home account which, although not currently
enforced, should not exceed about 10Gbytes and a project ( or spool area) that can be several Tbytes. The project
area can be access from $HOME/lustre . An area corresponding to your account name will be created upon
request.
It should be stressed that NO DATA ON SCIAMA IS BACKED UP.
The size of the project data area will be monitored. Old data may be deleted without warning.
PUSHING OR PULLING DATA
As the Sciama Login servers are Internet facing we have to maintain a high level of security. Consequently copying
data to and from the environment in some configurations is restricted. However we have tried to make things as
easy as possible.
Transfers are only possible using the ssh protocol. The method of transfer will depend on the nature of the data
being copied (discussed later in this document). When transferring data you usually have a push or pull option.
Sometimes security restraints can be circumvented by choosing the correct option.
These are the general rules:1.) From within the Sciama environment you can only initiate a transfer (either push or pull) to machines on
the University of Portsmouth network or the “Universe” and “Cosmos” machines in Cambridge. Initiating a
transfer to a machine at another University or on the Internet ( eg home ) is not permitted ( no outgoing
ssh connections allowed).
2.) From outside the Sciama environment you can initiate an ssh push or pull transfer provided the local
firewall configuration allows outgoing ssh connections. That is, from a remote machine you can push or
pull data to / from the Sciama environment.
Page 3 of 9
COPYING FROM THE COMMAND LINE
The table below shows three common methods of transferring data from the command line:Copy Method
Use
Scp from the command line
Usually used for transferring single files, could be
large gzipped tar files.
Sftp from the command line
Usually used for transferring directory trees. Works
in the same way as ftp but more secure.
Rsync from the command line
Usually used for transferring large directory trees.
Rsync can resume sending if it is interrupted during
the transfer.
SCP FROM THE COMMAND LINE
The SCP syntax is as follows. You can either push the data:scp <source-file-to-transfer> <username-on-sciama>@login<n>.sciama.icg.port.ac.uk:<destination>
or pull the data:scp <username-on-sciama>@login<n>.sciama.icg.port.ac.uk:<source-file-to-transfer> <destination>
Example:gdb34@universe:~> scp gadget-icg.tar [email protected]:/users/burtong
[email protected]'s password: xxxxxxxxxx
gadget-icg.tar
100% 471MB 12.4MB/s 00:38
SFTP FROM THE COMMAND LINE
Page 4 of 9
The SFTP syntax is as follows:sftp <username-on-sciama>@login<n>.sciama.icg.port.ac.uk
Example:gdb34@universe:~> sftp [email protected]
Connecting to login1.sciama.icg.port.ac.uk...
[email protected]'s password: xxxxxxxxx
sftp> get gadget-2.0.6.tar.gz
Fetching /users/burtong/gadget-2.0.6.tar.gz to gadget-2.0.6.tar.gz
/users/burtong/gadget-2.0.6.tar.gz
100% 21MB 10.7MB/s 00:02
The following sftp commands are available:sftp> help
Available commands:
cd path
lcd path
chgrp grp path
chmod mode path
chown own path
df [path]
help
get remote-path [local-path]
lls [ls-options [path]]
ln oldpath newpath
lmkdir path
lpwd
ls [path]
lumask umask
mkdir path
progress
put local-path [remote-path]
pwd
exit
quit
rename oldpath newpath
rmdir path
rm path
symlink oldpath newpath
version
!command
!
?
Change remote directory to 'path'
Change local directory to 'path'
Change group of file 'path' to 'grp'
Change permissions of file 'path' to 'mode'
Change owner of file 'path' to 'own'
Display statistics for current directory or filesystem containing 'path'
Display this help text
Download file
Display local directory listing
Symlink remote file
Create local directory
Print local working directory
Display remote directory listing
Set local umask to 'umask'
Create remote directory
Toggle display of progress meter
Upload file
Display remote working directory
Quit sftp
Quit sftp
Rename remote file
Remove remote directory
Delete remote file
Symlink remote file
Show SFTP version
Execute 'command' in local shell
Escape to local shell
Synonym for help
RSYNC FROM THE COMMAND LINE
Page 5 of 9
Rsync can resume sending at the last file if it is interrupted during the transfer . For that reason, it's best to send a
number of smaller files rather than a single large file, as it can only start at the beginning of whole files. Again you
can push or pull the data.
The Rsync push syntax is:rsync -ravP -e ssh <tree-to-copy> <sciama-username>@login<n>.sciama.icg.port.ac.uk:<destination>
Example:gdb34@universe:~> rsync -ravP -e ssh Gadget-2.0.6 [email protected]:/users/burtong
[email protected]'s password: xxxxxxxxx
building file list ...
215 files to consider
Gadget-2.0.6/
Gadget-2.0.6/AUTHORS
318 100% 0.00kB/s 0:00:00 (xfer#1, to-check=213/215)
Gadget-2.0.6/COPYING
……..
….
…
Gadget-2.0.6/ICs/lcdm_gas_littleendian.dat
1966376 100% 6.92MB/s 0:00:00 (xfer#207, to-check=0/215)
sent 33472864 bytes received 4622 bytes 1366428.00 bytes/sec
total size is 33454341 speedup is 1.00
In situation where it is not possible to initiate an Rsync transfer from a remote machine it is possible to use an ssh
proxy ( via another machine on the UoP network) in order to complete a transfer. This is beyond the scope of this
document.
Page 6 of 9
TRANSFERRING DATA USING A GRAPHICAL FIL E MANAGER
There are many graphical file managers available across the different platforms. For MAC users please see the
following link:http://www.openssh.com/macos.html
For Linux users this document will give an example using Nautilus. For Windows users there is a WinSCP example.
DRAG AND DROP USING NAUTILUS
When using a remote desktop (FreeNX) a graphical file manager like Nautilus can be used from within the Sciama
environment:-
For UoP users you can mount your N: drive using myfiles.port.ac.uk . This should be used to transfer files to and
from your home environment. See:http://www.is.port.ac.uk/staff/workanywhere/fileaccess/connecttoserverlinux
Page 7 of 9
DRAG AND DROP USING WINSCP
For Windows the WinSCP program is a free download from:http://winscp.net/eng/download.php
From a remote Windows desktop a SFTP connection can be made to Sciama:-
Once connected the two pane file manager allows you to drag and drop:-
Page 8 of 9
TRANSFERRING FILES USING EMAIL (PORTSMOUTH USERS)
For smaller files you can use the web version of Groupwise. Email an attachment to yourself then enter
http://webmail.port.ac.uk in Firefox. Save the attachment to the filesystem.
Page 9 of 9
TRANSFERRING FILES USING MIDNIGHT COMMAND
GNU Midnight Commander is a directory browser/file manager for Unix-like operating systems. It is useful if you
only have access to an ssh shell (rather than the FreeNX remote desktop). It is installed on Sciama under the
modules facility. To access it use:
module add apps/mc/4.6.1
or

module initadd apps/mc/4.6.1
From the command line type “mc”:-