Distributing workers evenly between jobs

[jturner]

I have a problem with distribution of jobs across the farm and I am wondering if there is a feature in Qube that can be used to solve it:

My problem is that when we have multiple jobs rendering on the renderfarm one of the jobs invariably takes all of the workers and all the other jobs have to wait for the first job to completely finish. This is fine if all the jobs take the same amount of time to process, but if the slowest job gets in first (and we don't necessarily know which is the slowest job before we begin rendering, so we can't set it to a lower priority) then it can hog the farm for the whole day, without any of the other renders even beginning.

We could try limiting the CPUs or assigning each job to a group of workers, but the problem with these approaches is that as the quicker jobs complete I'd like to be able to assign the free workers to the other jobs that are still running - limiting CPUs or using groups fixes a job into a set number of workers for its whole run.

Is there a way to get Qube to divide the free workers on the renderfarm (or in a group) equally between the jobs using it (without letting any one job take all of the workers), but at the same time, when one of the jobs completes it would allocate each of the remaining jobs a share of the newly freed workers?

I hope I've been able to explain this clearly - please let me know if there already exists something that i can use to resolve this issue, or if something to create this behaviour could be included in a future release.

[shinya]

Hi there jturner,

Thanks for your question and the clear explanation of your situation.

Qube isn't designed to stop subjobs of a given job in order to jumpstart
a job that's of the same priority.  If you require that your shorter jobs to
start sooner, you'll want to give them higher numerical priority so that the
system preempts the lower priority, presumably longer-running, jobs.

You can submit jobs at different priorities, but you can also adjust the
priority of jobs that are already in the system, by accessing the Qube GUI's
right-click menu's "modify" option on a job so if you find that your shorter
jobs are being "blocked" by your already-running longer jobs, then you can
manually adjust the priority of your jobs.

[dmeyer]

jturner,

I believe the clustering feature would work well for you.  Say if you have 100 workers, and on a typical day you have 4 jobs in the queue, you can divide the 100 into 4 groups of 25.  Then submit each job to one of the four groups.  If Job 1 finishes faster than jobs 2,3,and 4, the Job 1 machines will then grab frames from those jobs.

[martyfitz]

Huh.  I came here trying to figure out the same thing.

Using your example, I was thinking along the lines of always dividing the workers evenly among the pending jobs until you get less than, say, 10 workers per job.  Then it's first come first served.  This would account for a wide dynamic range in the number of computers being workers, but not allow anyone to hog the whole farm. 

Something like: # of workers per job =  max(10, # of workers/# of pending jobs).  Not sure how to do that with Perl.

I'm trying to figure out the clustering.  Seems like it would take supervisor intervention if a number of jobs were submitted to the same cluster while other clusters were free.

Marty