Managing Storage to Local Machines

[chrisWhite]

I have a farm of 30-45 active render nodes and my file server running as our supervisor.  While our network should be one of the fastest networks in CO in reality we have university policy pushed down on our farm that brings our network to less-than-ideal speeds.  When we were using Muster as our renderfarm management solution it would distribute all of the Maya project files to each computer at the beginning of a job and then when it is finished it would do (a terrible job of) cleaning up those files on the completion of the job.

Is there a way to have Qube do this?  Right now I have it auto-mounting our server as a NAS that we can render off of but I'm worried that it will slam our network once we start pushing some really heavy jobs on it.  I've also tried manually copying the files to each computer and it works great but kind of defeats half the purpose.

Thanks!

[eric]

There isn't really a built-in capability for prepping a host like you've described. The Qube model does presume that you have the capacity and bandwidth to support for each Worker direct access of network storage.

You would want to look to one of two strategies for dealing with this issue:

• Running a prep job that copies the files, followed by a dependent job that does the render. The difficulty lies in making sure that all the Workers are prepped (the host_list flag will run a single job all hosts on the farm, or a specified list of hosts) so that the jobs could run correctly.
• The second approach would require modifying the backend of the Job Type (located in jobtypes/typename so that the initialization phase includes a copy of the file. Cleanup would happen in the cleanup stage at the end of the job. This wouldn't be too hard a modification as the code is well-structured.

That being said, I'm not sure that copying the files really buys you anything over direct access. If you have all the Workers copying the file, you'd probably consume the same bandwidth anyway.

[chrisWhite]

Thanks Eric!

I'll discuss this with my supervisor Monday and see what he thinks.  I'm not sure how quickly we'll be able to test it, our trial license just expired and I'm told our purchase is moving through the university bureaucracy but I would appreciate it if you left the thread unlocked for further questions once I can do more testing on heavier scenes.