QB_Boot Sucking Life out of my Qube Bliss *(Qube 5.50)

[mdonovan]

We sporadically get this error on our workers ....

----------------------------------
-     Worker 484.10
----------------------------------
Can't locate qbboot.pm in @INC (@INC contains: C:\Program Files (x86)\pfx\jobtypes C:\Program Files\pfx\jobtypes C:\Program Files\pfx\qube\\types C:\Program Files\pfx\qube\api/perl/qb/blib/arch/auto/qb C:\Program Files\pfx\qube\api/perl C:/Perl64/site/lib C:/Perl64/lib .) at C:\Program Files\pfx\qube\api/perl/qb.pm line 66.
BEGIN failed--compilation aborted at C:\Program Files\pfx\qube\api/perl/qb.pm line 66.
Compilation failed in require at C:\WINDOWS\Temp\job\0\484\484_10.pm line 6.
BEGIN failed--compilation aborted at C:\WINDOWS\Temp\job\0\484\484_10.pm line 6.
6.

 

[jburk]

does this only happen on certain workers?

[mdonovan]

it happens on random jobs ... and when it does it is on all the workers.

[jburk]

Do the random jobs have the 'export_environment' flag set, and are they from a consistent set of users?

I'm wondering if something in 1 or more user's environments is mangling the runtime environment on the workers.

[mdonovan]

We still have not solved this one ...

It seems to happen anytime submit a job while there is a job already rendering.

here are some excerpts from the logs ... help =/


PROPERTIES

Status
status      : failed
timesubmit  : 2009-12-08 07:44:57
timestart   : 2009-12-08 07:45:01
timeelapsed : 0:01:06
timecomplete: 2009-12-08 07:46:07

Basic Job Properties
id          : 573
name        : change_textures_heavy_b
prototype   : xsi
user        : Smoke
priority    : 50
cpus        : 13
tasks       : 100 (100 pending)

Advanced Properties
hosts       :
groups      :
omithosts   :
omitgroups  :
cluster     : /
requirements:
reservations: host.processors=1
restrictions:
dependency  :
dependsup   :
dependsdown :

Package
appversion      : 8.0.249.0
passes          : color
renderMode      : render
scenefile       : \\isilon.smoke.nyc\jobs\2009_Jobs\temp_development_sm99999\3D\PRODUCTION\xsi\xsi_development\Scenes\rendering\change_textures_heavy_b_qube.scn
verbose         : 1
xsibatchExecutable: C:\Softimage\Softimage_2010_SP1_x64\Application\bin\XSIBatch.bat

Notes


Details
account     :
agendastatus: pending
datetime_supervisorQuery: 2009-12-08 07:48:46
domain      : .
flags       : 2056
flagsstring : auto_mount,disable_windows_job_object
hostorder   :
kind        :
label       : qube1
lastupdate  : 2009-12-08 07:46:07
mailaddress :
pgrp        : 573
pid         : 1
retrysubjob : -1
retrywork   : -1
subjobstatus: complete
timeout     : -1

////// StdErr for One of the workers
----------------------------------
-     Worker 573.0
----------------------------------
Can't locate qbboot.pm in @INC (@INC contains: C:\Program Files (x86)\pfx\jobtypes C:\Program Files\pfx\jobtypes C:\Program Files\pfx\qube\\types C:\Program Files\pfx\qube\api/perl/qb/blib/arch/auto/qb C:\Program Files\pfx\qube\api/perl C:/Perl64/site/lib C:/Perl64/lib .) at C:\Program Files\pfx\qube\api/perl/qb.pm line 66.
BEGIN failed--compilation aborted at C:\Program Files\pfx\qube\api/perl/qb.pm line 66.
Compilation failed in require at C:\WINDOWS\Temp\job\0\573\573_0.pm line 6.
BEGIN failed--compilation aborted at C:\WINDOWS\Temp\job\0\573\573_0.pm line 6.
6.

[jburk]

This was tracked down to a single mis-configured host burning through all the subjobs.

Note to users: if you're getting frequent failures, it's often informative to sort the subjobs for a failed job by state and see if the hostname is common across all failed subjobs.

In this case, all subjobs for the bad worker were marked as complete, but processed no frames (visible in the 'Timeline' graph).