Author Topic: Making sure jobs of a certain type run alone on a worker?  (Read 3240 times)

chardavo

  • Full Member
  • ***
  • Posts: 11
Making sure jobs of a certain type run alone on a worker?
« on: November 17, 2009, 03:48:49 AM »
Hi!
I would like to be able to specify as a 'requirement' that a subjob (of a certain type) should (when booked) be the only one to run on a machine. I was hoping I could do this with the "reservations" flag, and be able to specify something like "host.processors=ALL" to make sure:
- it waits for a worker to be completely available before being booked to it
- when it's booked, it "reserves" all the procs to make sure nothing else gets booked to it until it's done.

I don't want to "lock all procs except one" on the workers obviously, because then ALL jobs would always run alone, not just the jobs of the type I care about. Also, machines have different numbers of procs (some 2, some 4, some 8) so I can't reliably hardcode a number (unless it's fine to go overboard.... is this ok: job['reservations'] = "host.processors=100" ?).

Any suggestions on how to flag a job to behave like this?
Note: these jobs are being submitted through python (if that makes a difference)

chardavo

  • Full Member
  • ***
  • Posts: 11
Re: Making sure jobs of a certain type run alone on a worker?
« Reply #1 on: November 17, 2009, 03:53:10 AM »
An idea would be to use a 'global', say global.specialTypeIsRunning. And then when submitting a job of that special type, it would set:
job['requirements'] = "global.specialTypeIsRunning=0"
job['reservations'] = "global.specialTypeIsRunning=1"

Would something like this work? Last time I tried using a 'global' variable, I believe an error occurred because the specified global variable didn't exist yet when it was checking it for requirements, or incrementing it (one or the other, I forget). I deducted it required adding it to all the worker config files, which is not something I want to modify. This may have changed, though.

jburk

  • Administrator
  • *****
  • Posts: 485
Re: Making sure jobs of a certain type run alone on a worker?
« Reply #2 on: November 17, 2009, 03:28:00 PM »
You can control this behavior with a combination of job requirements and reservations:

To ensure a job only runs on a empty host:

  qbsub -req 'host.processors.used eq 0' -reserv 'host.processors=1+' sleep 60


To ensure a job only runs on a empty host with greater than 1 proc installed:

  qbsub -req '(host.processors.used eq 0) and (host.processors.total gt 1)' -reserv 'host.processors=1+' sleep 60

chardavo

  • Full Member
  • ***
  • Posts: 11
Re: Making sure jobs of a certain type run alone on a worker?
« Reply #3 on: November 17, 2009, 04:35:23 PM »
That sounds perfect -
Awesome, thanks!