PipelineFX Forum

Qube! => Developer Customization => Topic started by: chardavo on November 17, 2009, 03:48:49 AM

Title: Making sure jobs of a certain type run alone on a worker?
Post by: chardavo on November 17, 2009, 03:48:49 AM

Hi!
I would like to be able to specify as a 'requirement' that a subjob (of a certain type) should (when booked) be the only one to run on a machine. I was hoping I could do this with the "reservations" flag, and be able to specify something like "host.processors=ALL" to make sure:
- it waits for a worker to be completely available before being booked to it
- when it's booked, it "reserves" all the procs to make sure nothing else gets booked to it until it's done.

I don't want to "lock all procs except one" on the workers obviously, because then ALL jobs would always run alone, not just the jobs of the type I care about. Also, machines have different numbers of procs (some 2, some 4, some 8) so I can't reliably hardcode a number (unless it's fine to go overboard.... is this ok: job['reservations'] = "host.processors=100" ?).

Any suggestions on how to flag a job to behave like this?
Note: these jobs are being submitted through python (if that makes a difference)

Title: Re: Making sure jobs of a certain type run alone on a worker?
Post by: chardavo on November 17, 2009, 03:53:10 AM

An idea would be to use a 'global', say global.specialTypeIsRunning. And then when submitting a job of that special type, it would set:
job['requirements'] = "global.specialTypeIsRunning=0"
job['reservations'] = "global.specialTypeIsRunning=1"

Would something like this work? Last time I tried using a 'global' variable, I believe an error occurred because the specified global variable didn't exist yet when it was checking it for requirements, or incrementing it (one or the other, I forget). I deducted it required adding it to all the worker config files, which is not something I want to modify. This may have changed, though.

Title: Re: Making sure jobs of a certain type run alone on a worker?
Post by: jburk on November 17, 2009, 03:28:00 PM

You can control this behavior with a combination of job requirements and reservations:

To ensure a job only runs on a empty host:

  qbsub -req 'host.processors.used eq 0' -reserv 'host.processors=1+' sleep 60


To ensure a job only runs on a empty host with greater than 1 proc installed:

  qbsub -req '(host.processors.used eq 0) and (host.processors.total gt 1)' -reserv 'host.processors=1+' sleep 60

Title: Re: Making sure jobs of a certain type run alone on a worker?
Post by: chardavo on November 17, 2009, 04:35:23 PM

That sounds perfect -
Awesome, thanks!