Introduction to iRODS
10th Oct 2014
Christine Staiger
SURFsara
Amsterdam, 8th May 2014
Contents
iRODS Concepts
The Basics
① Installation
② Tools for administration
③ Adding data and metadata
iRODS Concepts
3
What is iRODS?
4
What does iRODS provide?
• Storage virtualization of different disk and tape
storage systems
• A logical namespace across storage locations
• A rule engine to automate data management
according to defined policies
• Various client interfaces
5
iRODS Components
6
iRODS Rule Engine
7
The users view:
iRODS Data Collections
Collection
0
•
•
•
•
Similar to Unix Directory and Files
ichmod to set user/group ACLs
Core.re: msiSetDefaultResc(Resource)
.irodsEnv: irodsDefResource=Resource
Data
Object
Data
Object
Data
Object
Data
Object
A
Collection
1
…/Collection0/
…/Collection0/DataObject A-B
…/Collection0/Collection1/
…/Collection0/Collection1/DataObject X-Z
Logical Name Space
(iCAT) independent from
physical resources
Data
Object
Data
Object
Data
Object
X
8
In the background:
iRODS Resources
• (Storage) Resource is a Software or Hardware
system that stores data
• 3 Resource classes:
Cache
Resource
Low
Latency
Archive
Resource
High
Latency
Compound
Resource
POSIX
like
IO calls
9
iRODS Resource Groups
Resource
Group
Cache
Resource
Cache
Resource
Low
Latency
Low
Latency
@Server A
@Server B
Cache
Resource
POSIX
Low
like
IO Latency
calls
@Server C
10
Hands on The Basics
11
Demo Data Grids
Virtual machines: ssh user@IP
IP adress
User
Password
145.100.58.144
irods1
irods104
145.100.58.147
irods2
irods204
145.100.58.148
irods3
irods304
145.100.58.150
irods4
irods404
145.100.58.155
irods6
irods604
145.100.58.157
irods7
irods704
145.100.58.158
irods8
irods804
145.100.58.160
irods9
irods904
145.100.58.161
irods10
irods1004
Installation and configuration
• Execute ./irodssetup in the folder iRODS
• Export the icommands
export PATH=$PATH:/home/user/iRODS/clients/icommands/bin
• ~/iRODS/config/irods.config
# Database configuration
$DATABASE_TYPE = 'postgres';
…
$DATABASE_HOST = 'localhost';
$DATABASE_PORT = '5432';
$DATABASE_ADMIN_PASSWORD = 'irods2';
$DATABASE_ADMIN_NAME = ’alice';
# iRODS configuration
IRODS_HOME = '/home/alice/iRODS';
$IRODS_PORT = '1247’;
…
$IRODS_ADMIN_PASSWORD = ’alice';
$IRODS_ICAT_HOST = '';
.irodsEnv file
# iRODS personal configuration file.
#
# This file was automatically created during iRODS installation.
# Created Mon Jun 18 13:13:27 2012
#
# iRODS server host name:
irodsHost ’surfsara-cloud'
# iRODS server port number:
irodsPort 1247
# Default storage resource name:
irodsDefResource 'demoResc'
# Home directory in iRODS:
irodsHome '/alicesZone/home/alice'
The .irodsEnv
# Current directory in iRODS:
determines the zone
irodsCwd '/alicesZone/home/alice'
and the user for the
# Account name:
icommands-client.
irodsUserName ’alice'
You can only use one
# Zone:
such file at a time.
irodsZone ‘alicesZone'
iRODS i-commands
Documentation:
https://wiki.irods.org/index.php/icommands
(we don’t provide a exhaustive list of i-commands)
iRODS control
• Execute ./irodsctl <arg> in the folder iRODS
• Restarting, starting and stopping the iRODS and iCAT server
with
•
<arg> = restart, start, stop
• Restarting, starting and stopping only the iRODS server with
•
<arg> = irestart, istart, istop
• Initialise an account and log in with iinit
16
The icommands client
• icommands are automatically installed when you install the iRODS
server
• Users must install the client software to connect to an iRODS server
• Categories:
• Informational
• UNIX and FTP like
• Metadata
• Functional
ihelp
• ihelp
Lists all commands
• ihelp -h
Usage : ihelp [-ah] [icommand]
Display i-commands synopsis or a particular i-command help text
Options are:
-h this help
-a print the help text for all the i-commands
Run with no options to display a synopsis of the i-commands
iRODS Version 3.3.1
February 2014
ihelp
ienv
• Lists the iRODS environment variables.
• Equivalent to “iinit -l”
• Usage : ienv [-h]
NOTICE:
NOTICE:
NOTICE:
NOTICE:
NOTICE:
NOTICE:
NOTICE:
NOTICE:
Release Version = rods3.3.1, API Version = d
irodsHost=surfsara-cloud
irodsPort=1247
irodsDefResource=demoResc
irodsHome=/alicesZone/home/alice
irodsCwd=/alicesZone/home/alice
irodsUserName=alice
irodsZone=alicesZone
iuserinfo
• Show information about your iRODS user account or the entered
user
• Usage: iuserinfo [-vVh] [user]
name: alice
id: 10007
type: rodsadmin
zone: alicesZone
info:
comment:
create time: 01396887567: 2014-04-07.18:19:27
modify time: 01396887567: 2014-04-07.18:19:27
member of group: alice
member of group: public
ilsresc
• ilsresc lists iRODS resources and resource-groups
• Usage: ilsresc [-lvVhA] [Name]
alice@sara-cloud:~$ ilsresc
demoResc
alice@sara-cloud:~$ ilsresc –l demoResc
resource name: demoResc
resc id: 10010
zone: alicesZone
type: unix file system
class: archive
location: surfsara-cloud
vault: /home/alice/iRODS/Vault
…
Some commands
•
Unix-like:
ils, ipwd, icd,
imkdir, irm, icp
ichmod, ipasswd,
irsync, ichksum, imv
•
FTP-like:
iinit, iput, iget, iexit
•
Test: Create folder in the logical name space
ipwd
imkdir NewFolder
ils NewFolder
irm –r NewFolder
User and Data administration
•
•
Change to admin mode: iadmin
Quit admin mode: q
•
Make/delete a user:
mkuser, rmuser, moduser
(iadmin) mkuser user1 rodsuser
(iadmin) moduser user1 password *****
•
Parameters and their values: iadmin lt, iadmin lt user_type
•
iadmin mkresc, iadmin rmresc
•
Create new resource:
mkdir Data (in your home, not in the iRODS environment)
iadmin mkresc NewDataResc “unix file system” cache
145.100.57.139 “/home/alice/Data”
Put and replicate data
•
Add the file /home/login/put1.txt to resource
iput -v put1.txt
ils -l or ils -L
iput -h
•
ichksum put1.txt (checksum of copy in Vault)
md5sum put1.txt (checksum of file in /home/login)
•
iput -R NewDataResc put2.txt
ils -L
•
Replicate data: irepl -R NewDataResc put1.txt
•
Trim number of copies: itrim -S demoResc -N 1 put1.txt
Access Control Lists
• Change permission for a user:
ichmod read user1 put1.txt
List access: ils -A
• Change permissions for group “public”:
Make a directory in the iRODS environment: imkdir temp
Make directory readable for “public”: ichmod -r read public temp
•
ils -A
/bobsZone/home/bob/temp:
ACL - bob#bobsZone:own g:public#bobsZone:read object
Inheritance - Disabled
iRODS Metadata
•
Commands: imeta, iquest, idbo
•
Add metadata:
imeta add -d put1.txt “Date” “08.10.2013”
imeta add -d put1.txt “AnotherDate” “04.04.2014”
imeta ls -d put1.txt
•
Remove metadata:
imeta rm -d put1.txt “Date” “08.10.2013”
•
Querying the iCAT:
iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE
DATA_RESC_NAME like ’%Resc%’”
DATA_NAME = put1.txt
DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c
Federation between data grids
• List all existing zones: ils /
• Zone A acknowledges Zone B and vice-cersa
iadmin mkzone B remote Host:Port
• Zone A adds remote user from Zone B:
iadmin mkuser user#B rodsuser
• Example: Alice and Bob:
Alice acknowledges Bob’s zone:
iadmin mkzone bobsZone remote 145.100.57.74:1247
Bob acknowledges Alice’s zone:
iadmin mkzone alicesZone remote 145.100.57.66:1247
Alice creates remote user Bob:
iadmin mkuser bob#bobsZone rodsuser
Now Bob can list and alter files in Alice’s zone: ils /alicesZone
Data Replication between grids
• Bob copies a file from his zone to his account in Alice’s zone
irsync -R demoResc i:/bobsZone/home/bob/put1.txt
i:/alicesZone/home/bob#bobsZone/put1.txt
-R specifies the target resource
• Bob copies a file from his home directory to Alice’s zone
irsync -R demoResc put3.txt
i:/alicesZone/home/bob#bobsZone/put3.txt
• Bob copies a file from Alice’s zone to his home directory
mkdir test
irsync i:/alicesZone/home/bob#bobsZone/put3.txt
test/put3.txt
• Bob deletes a file in Alice’s zone
irm /alicesZone/home/bob#bobsZone/put3.txt
Thank you !
Appendix
Trouble shooting
ERROR: putUtil: put error for
/alicesZone/home/bob#bobsZone/put1.txt, status = -9000
status = -9000 SYS EXCEED CONNECT CNT
Host configuration problem on the server.
Solution: Edit iRODS/server/config/irodsHost
localhost you.rIP.XX.XX surfsara-cloud
localhost 127.0.0.1
./irodsctl restart