To run many jobs without having to start UCC each time, there is a batch mode.
Assume you have a bunch of job files in a directory 'jobs', then you can run them all through UCC as follows
ucc batch -i jobs
As job files, UCC will accept only files ending in ".u", ".jsdl" or ".xml".
You can run in "follow" mode, where UCC will watch the input directory, and will process new files as they arrive.
ucc batch -f -i jobs
UCC can also process JSDL files, to batch-process these, use the "-j" option:
ucc batch -i jobs -j
Getting the most performance out of UCC and your Grid installation can be a challenging task. Sending too many jobs to a site might decrease throughput, sometimes the client machine can be the limiting factor, etc.
You should experiment a bit to get the best performance for your specific setup. UCC has many options available for tuning. Here is an overview.
option (short and long form) | description |
-K,--keep | Do not delete finished jobs on the server. By default, finished jobs are destroyed. |
-m,--max MaxRunningJobs | Limit on jobs submitted by UCC at one time (default: 100) |
-t,--threads NumThreads | Number of threads to be used for processing (default: 4) |
-u,--update UpdateInterval | Minimum time in milliseconds between status requests on a single job (Default: 1000) |
-R,--noResourceCheck | Do not check if the necessary application is available on the target system (will increase performance a bit) |
-X,--noFetchOutcome | Do not fetch standard output and error |
-S,--submitOnly | Only submit the jobs, do not wait for them to finish |
-M,--maxNewJobs | Limit the number of job submissions (default: 100) |
-s,--sitename | Specify which site to use |
-W,--siteWeights | Specify a file containing site weights |
-j,--jsdl | Assume jobs are in JSDL format instead of the default JSON .u files |
By default, the UCC batch mode will select a random site for running a job. You can modify the selection in different ways.
Say you have two sites where one site is a big cluster and the other a small cluster. To send more jobs to the big cluster, you can use the site weights file,
#example site weights file for use with "ucc batch -W ..." BIG-CLUSTER = 100 SMALL-CLUSTER = 10 #send no jobs to this site BAD-CLUSTER = 0 # set default weight (for any sites not specified here) UCC_DEFAULT_SITE_WEIGHT = 10
This would tell UCC to send 10 times more jobs to the "BIG-CLUSTER" site, and send no jobs“to the "BAD-CLUSTER". All other sites would get weight "10", i.e. the same as "SMALL-CLUSTER".