Author Topic: Submission woes...  (Read 6825 times)

freedomfries

  • Jr. Member
  • **
  • Posts: 4
Submission woes...
« on: February 07, 2012, 03:05:15 AM »
Sorry for the newb questions,

 I'm not new to rendering, having used a dozen renderfarm managements systems, but I am new to Qube. Our studio recently deployed it to our farm and naturally, there's stuff to smooth out, learning curves, etc... We're using Maya with Vray.

This may be a confusion in semantics, but, even after having read the manuals, I'm not clear on what a Subjob represents in the submit Job basics section...

For example, Let's say I submit a 300 frame job with a subjob setting of 10. It seems that the render farm is running 10 frames at a time. So even if I have 50 nodes, only 10 nodes will pick up the job, and the 40 remaining will pick up something else.

Is that what a subjob is?

If so, how do I configure each node to render a batch of say 5 frames of a particular job, so that the scene is only loaded once to render 5 frames on a particular node... Can't find this in the submit pag

Also, under Qube Worker Selection, Reservations, there is a string "host.processors=1". I'm not sure what that does and whether its necessary.

Agaon, sorry for the newb questions here, unfortunately there aren't any users familiar with the system here as we're rolling it out. The documention covers just about every option, but, in spite of my experience, I find the writing rather ambiguous.


jburk

  • Administrator
  • *****
  • Posts: 493
Re: Submission woes...
« Reply #1 on: February 07, 2012, 08:57:41 PM »
Qube and Maya (Dynamic Allocation)
First of all, if you're using the Maya jobtype, you don't need to think in terms of rendering in batches to minimize the number of times Maya is started and the scenefile is opened.

Once of the cool things Qube does with Maya is for each subjob, Maya is started only once in batch  "prompt" mode, and then we fire MEL commands at Maya mel: prompt.  The first command is to load the scenefile.  Once the scenefile is loaded, the worker then turns around and asks the supervisor for a frame to render; the supervisor sends just the number for the next unrendered frame.  The worker then fires a MEL command at the prompt instructing Maya to essentially move the timeslider (even though there's no UI) to the frame number in question, then when Maya replies saying it's done that, the worker tells Maya to render the current frame.  When the render's done, Maya tells the worker, and the worker asks the supervisor for another frame to render.

When a worker asks for the next frame but all the frames have been rendered, the supervisor tells the worker to shut Maya down.  Only once all the frames are rendered is the scene unloaded and a "quit -f" command fired at the MEL prompt.

The marketing term for this is "Dynamic Allocation".  Frames are dealt like cards to workers when they ask for the next frame.

So you get all the benefit of rendering in chunks (since Maya is started only once, and the scenefile and textures are loaded only once), but you don't have to decide in advance how big you want your chunks to be.  This also works out very nicely when you have a mix of machine speeds in your farm.  With chunks, the slower machines can take a lot longer to finish a 5-frame chunk than a faster machine.

With Qube's Dynamic Allocation, faster machines simply ask for another frame number more quickly than slower machines, so the faster machines usually end up rendering more frames than the slower machines, and all the workers usually finish up around the same time regardless of machine speed.

So a subjob in Qube is a single instance of Maya running on a worker.  You can start out rendering across 10 subjobs on your 50-node farm (to be polite and not plug the farm up for your fellow users), and if you decide part way through the job that you want to run on more (or all of the) nodes on the farm, you simply modify the job's "subjob process count" (job's "cpus" in older versions of Qube) to the number of copies of Maya you want, and more subjobs start up, more instances of Maya start running, and instead of 10 subjobs working through the frame list one by one, you now have 50 subjobs working through the list.

The list of frames to render is referred to as the "agenda" in Qube, and we often refer to frames as "agenda items" or "pieces of work", or simply "work".  Since Qube is used for many things other than rendering frames, the agenda can be comprised of a list of any type of work, but frames are the most common.

So your 300-frame job will create an agenda with 300 items in it (like a deck of cards with 300 cards in it), and you can increase the number of subjobs that process each card in the deck at any time.  Or shrink it, to give some capacity back to an emergency job, but Qube can handle this automatically with it's job priority.

Priority and Job Preemption
Now that you have a picture of how work is dealt out to subjobs like cards to players, you can get a good idea of how priority and preemption work in Qube.  Imagine a regular-priority subjob is running and rendering a frame.  A user submits a higher-priority job (or you bump the priority up on one of your other jobs).  The next time the subjob asks the supervisor for the next agenda item (the next frame), the supervisor will instead instruct the worker to surrender the job slot so that a  subjob from the higher-priority job can start in the slot instead.  The frames is finished, and the subjob from the lower-priority job goes back into a "pending" state, waiting for an available worker so that it can begin processing more frames at a later time.

This is "passive" preemption: finish what you're doing and then get off the worker.  Qube can also implement "aggressive" preemption (which approach to take is defined on the supervisor), which would kick the lower-priority subjob and frame off the worker immediately, throwing out the work done so far, and puts the frame and subjob back into a "pending" state, to be restarted when a worker is available.  Aggressive preemption can throw out a lot of cpu time; it's only really recommended in scenarios where all of the work in the farm runs very quickly, so tossing 10 seconds of compute time isn't a killer.  Last thing you want to do is toss 15 hours of a 16-hour render...  Farms that do nothing but transcoding or compositing are good candidates for aggressive preemption; farms that render anything longer than 30 second frames should stick with passive preemption.

Frame Chunking
If you're running jobs through Qube that don't take advantage of Dynamic Allocation, we support and make it easy to build job chunks.  In each submission interface in the QubeGUI, there's an "Execution" control in the Frame Range section.  The default is "Individual Frames", but you can select "Chunks with n frames" and set the chunksize, which will give you some number of chunks all of the same size.  If you know how many chunks you want to end up with and don't care about the chunk size ("I want to evenly spread 378 frames across 17 subjobs"), you can select "Split into n partitions", and this is where you'd get 17 chunks of near-equal size.  It does the arithmetic for you, so you don't end up with 18 chunks instead with the last chunk being 1 frame (which always used to happen to me...)


Workers and job reservations
The reservation string host.processors=1 determines how many worker job slots to "consume" while a job is running.  Workers advertise having a certain number of "job slots" available when they boot; this is defined by the worker_cpus setting for the worker.  The default value is 0; each worker will advertise as many job slots as it has cores installed.

When you have an 8-core worker with 8 job slots defined, and you submit a job with the reservation set to host.processor=1, you're saying that up to 8 of these subjobs could fit on a single worker.  So if you wanted each subjob to run all by itself on a worker, you could either set the processor reservation to 8, or use the shorthand reservation host.processors=1+, which means "start on a worker with at least 1 free slot, but reserve them all as soon as I start".  A worker with a 1+ reservation job running on it will show all slots in use, as in 8/8.

If every job you send to all your workers are going to be multi-threaded and expected to consume the entire worker, it's common to set each worker as advertising only 1 job slot with worker_cpus=1.  Then, with the default 1-slot reservation for each job, it's always 1 job to 1 worker.
« Last Edit: February 07, 2012, 08:59:23 PM by jburk »

freedomfries

  • Jr. Member
  • **
  • Posts: 4
Re: Submission woes...
« Reply #2 on: February 07, 2012, 10:58:23 PM »
Thanks Jburk for a well explained breakdown of the dynamic allocation system. I like it and think it's clever.

So regarding job reservations, "workers" is yout term for render node, am I correct?


and setting a host.processor=1 value means that each worker, regardless of how many cpus/cores, will handle only one frame, or subjob, at a time? Does that meant that only one cpu or core is being reserved to render on, or that one subjob only will be using all cpu resources on that worker?

I really appreciate your help, this is helping me figure out how to troubleshoot our farm and use it accordingly.

jburk

  • Administrator
  • *****
  • Posts: 493
Re: Submission woes...
« Reply #3 on: February 08, 2012, 11:58:28 PM »
Qube has clients (which submit), workers (which execute jobs - a render node), and the supervisor.

A machine can be both a client and a worker at the same time.  It's bad form to make a supervisor also be a worker, too heavy a load...

Workers have "job slots" (think slots in a toaster).  A worker can be configured for 1 slot, as many slots as it has cores (the default), or in special cases more slots than it has cores.  It's completely arbitrary.

A job has an agenda (the list of frames or chunks to render), but it also has a subjob list.  Each subjob will always process one agenda item at a time.  You don't need as many subjobs as you have frames, each subjob can process multiple frames sequentially; it just asks the supervisor for another frame when it's done the one it's working on.

The job reservation string defines how many slots on a worker a single subjob will reserve (consume) when it is running.  With an out-of-the-box default job reservation of host.processors=1, each subjob reserves 1 worker slot, and an 8-core host with 8 job slots will run 8 separate copies of Maya, with each copy will be rendering a single frame.

If you set the job reservation to host.processors=4, each subjob will reserve 4 worker slots, and an 8-slot worker will run 2 subjobs at the same time.

The Maya jobtype has a 'renderThreads' control in the Renderer section of the submission UI.  The default value of -1 (which is an invalid number of threads) is a clue to the code that runs the job on the worker to set Maya's internal number of threads to the same number of worker slots in the job reservation string.

So if you leave renderThreads = -1, and set the reservation to "host.processors=4", each subjob will reserve 4 worker slots and run a 4-threaded render in Maya.  If you look through a subjob's stderr logs, you will set the "setattr renderGlobals.numCpusToUse = 4" MEL command.

The flip side to this is if you set renderThreads=8, but forget and leave the reservation at 1 slot (host.processors=1), you will get 8 subjobs on an 8-slot box, each running an 8-threaded render.  64 render threads on an 8-core box.  Ouch: "Why are my renders running so slowly?"

If you use the "host.processors=1+" syntax in your reservation, you should set the renderThreads to the number of cores on a worker; in this case, an 8-slot worker will get 1 subjob dispatched to it that will run an 8-threaded render...

If you've been using Maya for a long time, you know that you can set Maya's 'numCpusToUse' renderGlobal attr to 0 as a shorthand for "use all available cpus".  This only works for the Maya software renderer, and is not well supported by mentalray or V-Ray; I think mr will give you 2 threads, and v-ray will probably decide to render in 1 thread.  (Anyone know the answer to this?)  If mentalray understood the '0', you could set the job's renderThreads to 0, and the reservation to 1+, and then each worker would run 1 subjob that would automatically run as many threads as there are cores.  But it doesn't, and this can be a problem on farms with a mix of 4-, 8- and 12-slot worker.

In that case, it's best to submit all your renders as 4-threaded; renderThreads=-1 and reservation of host.processors=4.  A 4-slot box will run 1 subjob, a 8-core will run 2, and a 12-core will run 3 subjobs.

And remember, you can always add more subjobs to a job.  You can't remove subjobs once they're added, but you can "complete" or "retire" them.  When you mark a subjob as complete, it will finish the frame it's working on, then exit and go into a "complete" state.  You can always put it back to work by selecting the individual subjob in the QubeGUI and "retry" it with rmb->Retry.

« Last Edit: February 09, 2012, 12:02:50 AM by jburk »