Instructions for network computing
resources in Institute of signal
processing
1) Use program jobitc
When you run your programs in the Lintulas machines, use program jobitc. The program jobitc is
used for distributing and sending jobs over Lintulas Linux machines. There is a tutorial in the
internet how to use it, http://www.cs.tut.fi/~jobitusr/jobitc.html. There is told for example how to
run Matlab programs with jobitc, or how you can run multiple jobs at the same time. When using program jobitc, the load is distributed equally between machines, so programs are
ready quicker. The program jobitc also closes Matlab automatically after use, in which case Matlab
licenses are freed as they should.
Servers
Personnel in Institute of signal processing are free to use jobitc on every Lintulas Linux machines
where they have ssh­access and where jobitd­daemon is ran. The machines are
•
Classes TC415 and TC407 (jobit groups tylli and kaija)
•
Audiogroups machines (jobit group arg)
•
Outolintu cluster (jobit group outolintu)
Groups tylli and kaija can be used freely with program jobitc by personnel in Institute of signal
processing. Using machines of Audio group, as well as Outolintu cluster, is restricted. Arg­
machines are directed to personnel in audio group. In Outolintu cluster it is meant to run Matlab
programs which needs a lot of memory. You can get ssh­access to Outolintu cluster by filling an
application (<linkki hakemuslomak­kee­seen>) and returning it to Juha Peltonen (room TC426).
Using jobitc
Program jobitc is found from every Lintula machines, when you add its path /share/jobit/bin to the
PATH­variable. It can be done as follows:
Shell
Command
Configuration file
bash
export PATH=$PATH:/share/jobit/bin
.profile
tcsh
set PATH ($PATH share/jobit/bin)
.login
zsh
export PATH=$PATH:/share/jobit/bin
.zprofile
Depending which shell you are using, add the right command to its configuration file, if you like to
have the path always in your use. This can be done with command 'echo “<command>” >>
<config. file>'. For example, in bash shell the command is
echo “export PATH=$PATH:/share/jobit/bin” >> ~/.profile
Noora Nurminen, Version 0.6 03/17/06
Your shell can be found out for example with command “echo $SHELL”. If you can't set the PATH
variable, contact to Lintula administrators ([email protected]).
After adding the command to PATH variable, you got two new commands: jobitc and jobitstats.
Command “jobitstats <group>”, where <group> is some jobit group, lists the groups machines
status information on the screen with the same way as in web page http://www.cs.tut.fi/cgi­
bin/run/~jobitusr/servers.py. The most simple use of jobitc is to add “jobitc ­c” in front of your command.
Running program myProgram with parameters param1 and param2:
jobitc -c ./myProgram -param1 -param2
Running m­file myMfile.m with Matlab:
jobitc -c matlab myMfile
Running the following commands on Matlab. First loading mat­files myParams.mat variables var1
and var2, which are then added to variable out. The result out is saved to the mat­file
newParams.mat. Notice, that when running multiple commands separated with semicolons, all the
commands must be inside quotes:
jobitc -c matlab “load myParams; out=var1+var2; save newParams out”
Running Linux command “ps ­fu jobitusr” with program jobitc, and directing the output to file
ps_commands_output.txt. Notice, that quotas are needed again:
jobitc -c “ps -fu jobitusr > ps_command_output.txt”
Running multiple jobs and distributing jobs can be done by writing a jobit­file, where commands
are written everyone on its own line. More instructions to jobitc can be found from web,
http://www.cs.tut.fi/~jobitusr/jobitc.html.
2) Matlab
When using Matlab, it reserves a licence to itself and one licence to every used toolbox. There is
only limited amount of licenses. This causes problems, when users don't close Matlab immediately
after using it, and so all licenses reserved will stay reserved.
To make the Matlab licenses serve everybody, and make the Matlab work to researchers as well as
to students, users mut take care of next issues.
1. Close Matlab right
.
To release licenses after using Matlab, it must be closed by choosing from Menu 'Exit
MATLAB', giving command 'exit' to Matlabs commandline or with accelerator key Ctrl­Q.
When using program jobitc, the program takes care of opening and closing Matlab rightly.
2. Close Matlab aways, when you don't use it
.
When running m­files, closing Matlab can be done by adding command 'exit' to the end of the m­
file, so the Matlab will be closed after running the m­file. When using program jobitc, it takes
care of closing Matlab after the program has ran.
3. Especially long and/or heavy works should be done as batch processing with jobitc
. You can run your programs without constant monitoring. Write your work to m­file. If some
Noora Nurminen, Version 0.6 03/17/06
medial results are needed, they can be saved to mat­files with command “save <mat­file>
<saved variables>”, for example
save midresults var1 var2
% saves var1 and var2 to midresults.mat
Pictures behoves to save to files in place of plotting them. This can be done for example with
commands
figure('vis', 'off')
plot(rand(1,20))
saveas(gcf, 'invfig.fig', 'fig')
% Open figure with visualization off
% Plot something to it
% Save the figure e.g. as Matlab-figure
The picture can be saved also as PNG­ or JPG­formats. FIG­file can be opened in Matlab later on
with command openfig('invfig.fig', 'reuse', 'visible')
When using Linux machines, email can be used to track medial results or the program ending.
Next commands can be used to send email while or after processing:
system('echo Hello World | /bin/mail [email protected]');
% Sends message “Hello World”
system(['echo Fig',num2str(1+1),' is ready | /bin/mail [email protected]']); % Sends “Fig2 is ready”
system('/bin/mail [email protected] < output_file.txt'); % Prints output_file.txt to the message
More help for using Matlab commands can be found with command “help <command>”. For example
help figure
Between 8:00 am and 6:00 pm use only three (3) Matlabs at the same time
4.
Matlabs use is monitored and reserving too many licenses will be noticed and remarked. At night
after 6:00 pm to 8:00 am Matlabs use is freed. Restrict and supervice your usage. If you
suspect that your Matlab­jobs take longer than the night or you don't know the execution time,
you should ask promise with good reasons from Juha Peltonen ([email protected]).
Malpractice will cause a penalty.
You can stall the Matlab job so that it will begin after 6:00 pm. This can be done for example by
using command “sleep <seconds>” before launching Matlab. Sleep delays the command with
given time. The executed command is given to the same line separated with semicolon:
sleep 18000; jobitc matlab myMfile
# 18000 seconds is 5 hours
Matlab­work can be interrupted, variables saved to mat­file and Matlab closed in the middle of
execution, for example when clock gets over 8:00 am. This can be done as follows and the same
code can be sued when starting the program again from the point where it ended last time.
Change to the m­files for­loops next structure. Change necessary values (maximum amount of
iterations max_iter, conditions of for­loops and saved results <variables> in the end) so that they
fit in to your program. You can add the structure to many places to your prgram, but use then
different name for the mat­file (variables.mat). Add the structure especially to parts which you
think will take long time, but do not over react.
max_iter = 10;
iter = 1;
if exist('variables.mat','file')
load variables
% set max_iter
% set the start point in the beginning
% If some results already, use them
Noora Nurminen, Version 0.6 03/17/06
end
for i = iter:max_iter
iter = iter + 1;
% Go throught every iteration
% Heavy counting – Do something here
b_time = clock;
% Take the time variable
if (b_time(4) > 8) & (b_time(4) < 18) & (iter < max_iter) % If time = 8:00 – 18:00
save variables
% save all variables to file variables.mat
exit
% and exit Matlab.
end
end
% In the end:
save myResult <variables>
delete variables.mat
% Finished. Save necessary parameters to myResults.mat
% delete variables.mat, it's not needed anymore
The structure saves all variables to file variables.mat and interrupts when it's over 8:00 o'clock or
before 18:00 o'clock. When the program is ran again, it loads the variables from the mat­file and
starts from the point where it was before. In the end all important variables are saved to file
myResults.mat.
Do not reserve licenses over 24 hours
5.
If you suspect that your Matlab­run is going to take over 24 hours, ask promise from Juha
Peltonen ([email protected]). Malpractice will cause a penalty.
As in the previous example, also over 24 hours Matlab programs can be interrupted. This can be
done by changing the previous example as follows. Add to the beginning of your m­file command
b_time = clock;
% Take the time variable
and remove it from the for­loop. Change the condition of the if­statement “if (b_time(4) > 8) &
(b_time(4) < 18) & (iter < max_iter)” as follows.
If etime(clock,b_time) > 86400
% If 86400 secs (24 hours) since the program started
Important
•
When you use program jobitc for the first time, it creates a key for you. The key helps you to use jobitc. When creating the key, it creates a password to it also, which is double­checked. At the
same time the key is activated, so you need to type the password once again. When all necessary
groups are added to the key, you should not usually have to type passwords at all. A key
equipped with a password is totally safe, but a key without a password is not. So do not care
about the possibility “(empty for no passphrase)” given by the program when creating the key.
•
If you suspect that your program will run longer than 20 days, you must give to jobitc a
parameter “­lifetime <days>”, where <days> is the amount of days you think the execution time
is. Or else the monitoring of the job will stop and possible errrors are not detected.
Noora Nurminen, Version 0.6 03/17/06
•
If your Matlab program tries to plot something while running it with jobitc, it substantially slows
down the execution. Therefore change every real time plot to the structure which saves the
picture to a file.
•
If your program uses lot of memory, you can give a parameter “­mem <KB>” to jobitc. It tells to
jobitc how much at least you need memory to your work and then jobitc searches only machines
which have such amount of memory. For example, if you know that your program will use at
least 26 MB memory, you can give parameter “­mem 26000” to jobitc. Then machines which
have less 26 MB free memory will be discarded.
•
If you suspect any disruptions or you have proposals for improvement, please contact to Lintula
([email protected]) or Juha Peltonen ([email protected]).
Noora Nurminen, Version 0.6 03/17/06