In this report an ability of parallel Polar data processing using cluster of Centrum Informatyczne Świerk (hereafter CIŚ) has been presented. Chapter two contain a description of testing environment in chapter three details of parallel calculation described. Chapter four and five presents respectively example (sort of how to) of CIŚ cluster using in context of Polar data analysis and results of conducted tests. In appendix A some useful commands has been described.
2. Testing environment
Polar data testing calculation has been carried out in CIŚ cluster. The software used in mentioned analysis downloaded from official Polar software repository. Details about an access to Polar repository and about code compilation and needed library has described in previous report.
The data used in tests came from first phase of Polar detector tests. This data can be found on Swiss server
/RAID/polar/QM1/TestInit 20130829/data raw folder. The QM1 catalog contain 1075 files saved in binary format defined by CERN root framework (details about this format can be found at http://root.cern.ch/drupal/content/root-files-1). Those files contain 43 run of data acquisition, precisely 25 files
*.root per each run. For all 43 runs following sequence of processing had to be done: pedestal subtraction, cross talk correction, energy calibration an data merging. The process program which can be found in
/software/analysis/ProcessData/ in Polar repository is dedicated for mentioned sequence of analysis.
3. Parallel data processing
In a case of Polar software there is no possibility of parallel data processing in a level of single code instruction. This software does not support multithreading. One chance of doing parallel data processing is to divided whole input data set into smaller subsets and run for each one an instance of process program. The number of subsets is limited by a logic of Polar data processing. Taking this into account, I prepared python’s script called
splitter.py which takes as input parameters following variables: path to input set, path to place where output should be write and the number of subsets on which initial set should be divided. The
splitter.py script consist of three function. First one, called
readconfig() reads input parameters from
polarCIS.conf configuration file. The following format of configuration file has been adopted:
parameter name:value (see, Fig. 1.)
The second function called
prepdata() reads initial set of data and on the basis of names of
*.root files it determines the number of acquisition queue (runs). The number of queue is a very important parameters because it determines the maximum value of subsets. On the basis of number of queue and the number of subsets third function called
splittask() creates commands sequence which will call separate instances of procces program (one instance per one subset). Mentioned sequence of commands are writing into files (one file, one command) called ”
taskX.sh”, where X is a successive number of file (or subset). The
taskX.sh files will be a parameters for qsub command which requests resources of CIŚ cluster.
Let’s assume that we want to make an analysis of Polar data which are in mentioned QM1 catalog in four parallel processes. Following steps should be performed.
- 1. Set parameters in configuration
- 2. Run
splitter.pyby command python
As an output user should see information like this:
No of subsets: 3.0
No of runs: 43
For n-1 nodes: 14.0 runs
Last node: 15.0 runs
process all -t QM1 -r 0000 0013 -i QM1/ -o results_QM1_1 -ow
process all -t QM1 -r 0014 0027 -i QM1/ -o results_QM1_2 -ow
process all -t QM1 -r 0028 0042 -i QM1/ -o results_QM1_3 -ow
procPath : polar2/software/analysis/ProcessData/
splitter.py was run, four files (four because we decided to have four parallel process) should be created, namely,
task4.sh. This files should contained
instructions like this:
process all -t QM1 -r 0000 0010 -i QM1 -o results_QM1_1 -ow
- 3. Start task files on CIS cluster. To run tasks defined in
task1-4.shfollowing commands need to be used.
qsub -q short taks1.sh
qsub -q short task2.sh
qsub -q short task3.sh
qsub -q short task4.sh
If we want to avoid typing four times above commands we can place them into some shell file. Then save this file as for instance
chmod 755 run_tasks and run it by
5. Tests results
Method discussed in chapter two and three has been tested on CIŚ cluster. The time of processing data was measured and the results of measurements was shown in Fig.2. As can be seen from this figure the initial set of data was divided from two to fourteen subsets. For each of subsets Polar’s data has processed. The real and CPU usage time was measured. If we divided an initial set to two subsets the real time of data processing was equal about 140 minutes. When we divided it to fourteen subsets we was able to processed them in 8 minutes. For single CIŚ node when I did not use proposed
method of parallel analysis the time of whole process took about 2h05m. For comparison, Polar data processing was run on dedicated Swiss server and the time of whole analysis was order of 2 hours (precisely, 1h56m).
Below we present some useful commands which can be used during work with CIŚ cluster.
This command submit jobs on server. It has following options,
[-a date time] [-A account string] [-b secs] [-c checkpoint options] [-C directive prefix] [-d path] [-D path] [-e path] [-f] [-h] [-I ] [-j join ] [-k keep ] [-l resource list ] [-m mail options] [-M user list] [-N name] [-o path] [-p priority] [-P user[:group]]
[-q destination] [-r c] [-S path list] [-t array request] [-u user list] [-v variable list] [-V ] [-W additional attributes] [-X] [-z] [script]
Details about all above options can be found at http://www.clusterresources.com/torquedocs21/commands/qsub.shtml
qsub -q short task1.sh
This command submit a short queue for tasks dened in task1.sh file.
This command show status of jobs. It has following options
qstat [-f [-1]][-W site specific] [job identifier... | destination...] [time]
qstat [-a|-i|-r|-e] [-n [-1]] [-s] [-G|-M] [-R] [-u user list] [job identifier... | destination...]
qstat -Q [-f [-1]][-W site specific] [destination...]
qstat -q [-G|-M] [destination...]
qstat -B [-f [-1]][-W site specific] [server name...]
Details about qstat options can be found at http://www.clusterresources.com/torquedocs21/commands/qstat.shtml
qsub -u user name,
This command will show some useful information about jobs requested by a given user.
The qdel command deletes jobs. It has a following options,
|-p|-W |-t ] [ ]...| 'all' | 'ALL'
Details about qdel options can be found at http://www.clusterresources.com/torquedocs21/commands/qdel.shtml
Author: Artur Narwid