Author Topic: Rendering multiple passes in one job increases rendertime  (Read 8264 times)

laschmoove

  • Sr. Member
  • ****
  • Posts: 29
Rendering multiple passes in one job increases rendertime
« on: March 26, 2009, 11:30:20 AM »
Hi,

we are using qube with the XSIBatch job and we are experiencing the following:

If there are several passes (in this case more than 20) in one scene and we submit a job that should render all of them, the rendertime is dramatically higher than the sum of the rendertimes needed by jobs that only render a subset of the passes. Can anybody explain that to me or provide me a solution?
I suppose that it is a question of memory consumption, but I have no idea how to fix it. And it would be a pain for the artists to always submit 20 Jobs for each rendering.

Any help or idea is very appreciated.

Thanks,
laschmoove

shinya

  • Administrator
  • *****
  • Posts: 232
Re: Rendering multiple passes in one job increases rendertime
« Reply #1 on: March 27, 2009, 06:48:03 AM »

Are your workers multi-core systems using the default settings for jobslots?
For example, do you have 4-core machines with the default "worker cpus" setting, so that they can each run 4 subjobs at a time?  If that's the case, it may actually be the case that your workers are being oversubscribed, in terms of memory resource. 

Ideally, your users should all estimate, as accurately as possible, the amount of RAM that each job needs, and then add it to the "reservation" field of each job at submission, as in "host.memory=1024" (which means 1024MB), so that their jobs won't step on each other's toes-- i.e., that way, each job will "reserve" the eccessary amount of RAM to run smoothly, without some other large jobs running on the same worker, fighting for limited RAM.

If that's not possible, you may want to limit your workers to only run 1 subjob at a time, per worker, by setting their worker_cpus to 1.  That way, you'll know for sure that each worker will only run 1 instance of XSI at a time, and that should alleviate memory issues caused by running multiple subjobs at once.
By default, workers can run up to N subjobs at a time, where N = number of cores on the system.

Optionally, you can submit jobs with "host.processors=1+" (yes, that's the digit ONE, followed by a PLUS sign), so that each subjob from those jobs will occupy all jobslots on each worker, thereby preventing other jobs from running on the worker until they're done.


laschmoove

  • Sr. Member
  • ****
  • Posts: 29
Re: Rendering multiple passes in one job increases rendertime
« Reply #2 on: March 31, 2009, 11:57:08 AM »
Hi Shinya,

thanks for your answer. Unfortunately, we are already using qube! that way (allowing just one subjob on one worker at a time).

Do you have another idea?

Thanks a lot,
laschmoove