'completed' frames still marked as 'running'

[foureyes]

Hello!

I have installed qube 5.3 and render Maya 2008 and XSI 6.5 jobs submitted by qube jobtypes. I encounter the same problem in both software, which is for example, the normal rendering time for a particular job is 1 min, but for certain frames, it takes 30 mins to render but the status is still marked as 'running'. When I view the frame in Windows, it is actually finished. It seems that the host cannot communicate back to the supervisor to inform it that it has finished the job and change the status to 'complete'. What is the cause and solution to the problem?

Thanks.

foureyes

[eric]

This kind of problem can surface if the render process has run out of memory. If you are running several Maya or XSI subjobs on the same host, you run the risk of consuming most of the memory. If that happens, the process may thrash, making it difficult for the Worker to monitor and control it.
If you restrict the number of subjobs to 1/host, you may see an improvement in this problem. That could indicate that memory contention is causing the problem.

[cyborg]

Are there any logs file can identified which hosts are running out of memory?

[sosborne76]

So is it possible to run more than 1 subjob per host?

[Scot Brew]

So is it possible to run more than 1 subjob per host?

Yes, by default qube sets each Worker machine up to be able to run as many subjob processes as it has CPUs.  For most multi-threaded renderers (like Maya), locking down the Workers to only run 1 subjob process at a time is preferred. 

One can control the locking of Workers in the QubeGUI by right-clicking on the host, and doing a "Lock/Unlock..." and then specifying the number of subjob processes that the Worker can run.  Alternatively one can also override this information directly in the qb.conf and qbwrk.conf files.

[eric]

In answer to the question about logs, you can use the stats display in the GUI to get an idea of the memory consumption. Unfortunately, you may have to login to the machine while it's running and monitor the memory consumption directly.