DUCKS – Distributed
User-mode ChirpKnowledgeable Server
Joe Thompson
Jay Doyle
DUCKS Motivation
3
Usability
2
1
Performance
Chirp
&
Condor
DUCKS Goals
Bring together functionality of Condor and CHIRP in
an easy to use package.
Abstract Condor and CHIRP interfaces.
Intelligently distribute files over CHIRP servers.
Provide simple interface for the Chirp Active Storage
Program-To-Data model.
Provide simple interface for the Condor Data-ToProgram model.
DUCKS
Client
Incoming
Incoming
Incoming
Message
Message
Message
Queue
Queue
Queue
Chirp
Server
List
Transaction
Handler
MySQL
Chirp
Tracker
Transaction
List
Timeout
Handler
Garbage
Collector
ducks_put <username> <local_file> <ducks_name>
Local Machine
DUCKS
Server
1) Verify the <username/ducks_name>
pair is not already in the DB
Store_Request
<username> <ducks_name> <filesize>
2) Find a Chirp node with enough
free space to store the file
3) Query database for path name
to use on the Chirp node
4) Parse response to get the
Chirp storage location for
<local_file>
chirp_put
<local_file> <chirp_node> <path_on_node>
Store_success
<username> <ducks_name> <chirp_node>
<path_on_node>
Store_Response
<Chirp_node> <Path_on_node>
File
5) Update DB to reflect this
file storage
Chirp Node
ducks_get <username> <ducks_name> <local_name>
Local Machine
Get_Request
<username> <ducks_name>
2) Parse response to get the
Chirp storage location for
<ducks_name>
DUCKS
Server
1) Query DB for Chirp location of
<ducks_name>
Chirp Node
Get_Response
<Chirp_node> <Path_on_node>
chirp_get
<chirp_node> <path_on_node> <local_file>
File
ducks_delete <username> <ducks_name>
Local Machine
delete_Request
<username> <ducks_name>
DUCKS
Garbage Collector
Server
1) Set the delete_flag of the
<username/ducks_name> entry in
the DB
DUCKS
DB
<chirp_node> <path_on_node>
…
…
2) Periodically query the DB for
files with the delete_flag set
3) Delete files in the list
Result Set
Chirp Node
Delete File
4) Remove files from DB
ducks_ls <username> <search_string>
Local Machine
ls_Request
<username> <search_string>
DUCKS
Server
1) Query the DB for all files
owned by the user that match
“%<search_string>%”;
ls Thread
2) Start a background thread and
pass it the result set containing the
found files
Result
Set
3) Iterate through the
set and send the file
information to the
client
File_info
Client
Job Request
Wrapper/
Submit
Scripts
Distribution via Chirp
Active Storage to nodes
already storing input
Chirp Nodes
input01.txt
input02.txt
Response with
locations of
input files
input03.txt
ducks_get
exe/libs
request
DUCKS Server
input04.txt
Client
Chirp Nodes
Distribute tasks to
any available node
Condor
exe/libs/input_file
request
DUCKS Server
Get input and exe/lib files
from other nodes in the
Chirp cluster with ducks_get
Transfer Speed Comparison: DUCKS vs Standalone
Chirp (avg. 3 runs)
30
Transfer Speeds (MB/s)
25
20
15
Chirp
DUCKS
10
5
0
1 (x1000)
10 (x100)
100 (x10)
File Size (MB) (x # Files)
1000 (x1)
Execution Time Comparison: DUCKS vs Standalone
Condor (avg. 15 runs)
300
Execution Time (s)
250
200
150
100
50
0
Condor
DUCKS (JtoD)
Job Execution Method
DUCKS(DtoJ)
DUCKS Future Work
The basic framework is implemented.
Add a more robust file interface.
Implement DUCKS management of job status
information. (Queued, Running, Complete)