Qube issues after reimaging.

[Chris Computing]

I just took over the roll of Sysadmin at my company and built a brand new Windows 7 image from scratch. I put version 6.3.4 of the core, worker, and GUI on the workstations, via manual install. Our supervisor was left untouched fine and that server hasn't been touched.

BUT! Even though the supervisor sees all of the clients, and shows they are available and unlocked, every job fails that tries to render to the new computers.

Please help, I've gone over every doc and everything I could find and I have no idea why they are failing (all the configs are identical to the working ones in the labs I haven't imaged yet).

My test render is a plane with a cone, a torus and a cube sitting on it, a moving camera, and some mental ray materials and lighting, in Maya 2013. The computer's that have the Qube workers on it all have the same version of Maya. They also have Perl and Python 2.7.

Here is one of 2 errors I get (the other says it can't find perl, but doesn't always come up).

INFO: Using maya executable specified in submission
  [C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe]
Maya Binary: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe
maya::UniversalMayaRenderJob::initialize
maya::PromptMayaJob::initialize
maya::PromptMayaJob::mayaSetupWorkspace
INFO: setting MAYA_LOCATION to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt
to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt


INFO: timer of [300 secs] expired-- "nudging" maya for 'mel:' prompt at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 554.
ERROR: maximum nudge count

• reached on command []

timed out-- aborting! at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 569.
maya::MelProcessor::abort
rt

ERROR: exception caught in maya::UniversalMayaRenderJob=HASH(0x3208700)->run()
aborting at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 137.

INFO: reporting status [failed] to supe: qb::reportjob('failed')
')
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
INFO: HARNESS=[]
[]
ting status [failed] to supe: qb::reportjob('failed')
')
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
INFO: HARNESS=[]
[]

[jburk]

I would try remotely logging into the new worker as the same account that you're using as the proxy_account (if you're unsure, then you're using the default "qubeproxy" account, password is Pip3lin3P@$$wd).

Once Windows has created the desktop for that user, you should try starting the Maya GUI once in order to setup the MAYA_APP_DIR tree (the maya/scripts, etc directories in your user directory).

I'll bet that once you've done this, your jobs will run.

Before you attempt even a simple maya render, I'd submit a test job that only runs the command "set", to test whether the worker can accept and run a job, and send back the logs.

[Chris Computing]

Does it matter if I have 64 bit Perl or Python installed? Some of the machines say they can't find Perl and I have the 64 bit version on there.

[Chris Computing]

"'perl' is not recognized as an internal or external command,
operable program or batch file."

This is what I am getting now on all clients.

[Chris Computing]

ok, i submitted the just to to a few machines that are identically configured and this is what I get:

----------------------------------
-     Worker 1084.0
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.1
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.2
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.3
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.4
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.5
----------------------------------

$VAR1 = bless( {
  'hosts' => '',
  'priority' => 9999,
  'retrywork' => -1,
  'timesubmit' => 'Jan 11, 2013 8:07:58',
  'flagsstring' => 'auto_mount',
  'omithosts' => '',
  'lastupdate' => 'Jan 11, 2013 8:08:05',
  'pid' => 1,
  'timeout' => -1,
  'localorder' => 0,
  'groups' => '',
  'user' => 'cniew',
  'id' => 1084,
  'prototype' => 'maya',
  'mailaddress' => '',
  'restrictions' => '',
  'hostorder' => '',
  'retrywork_delay' => 0,
  'omitgroups' => '',
  'reason' => '',
  'name' => 'AVE-mray2013-2013test',
  'data' => '(=(scenefile=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest\\\\scenes\\\\rentest_v01.ma)(defaultRenderGlobals.currentRenderer=mentalRay)(rangeExecution=individual)(frameCount=120)(mayaExecutable=C:\\\\Program Files\\\\Autodesk\\\\Maya2013\\\\bin\\\\mayabatch.exe)(project=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest)(range=1-120)(renderDirectory=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest\\\\images))',
  'todotally' => {
    'complet' => 0,
    'killed' => 0,
    'running' => 0,
    'pending' => 0,
    'badlogin' => 0,
    'failed' => 0,
    'waiting' => 0,
    'unknown' => 0,
    'suspended' => 0,
    'blocked' => 0
  },
  'timestart' => 'Jan 11, 2013 8:08:01',
  'serverid' => 0,
  'agenda_timeout' => -1,
  'label' => 'qube1',
  'queue' => '',
  'pgrp' => 1084,
  'reservations' => 'host.processors=1+',
  'cpustally' => {
    'complet' => 0,
    'killed' => 0,
    'running' => 0,
    'pending' => 0,
    'badlogin' => 0,
    'failed' => 0,
    'waiting' => 0,
    'unknown' => 0,
    'suspended' => 0,
    'blocked' => 0
  },
  'status' => 'running',
  'subjobstatus' => 'pending',
  'dependency' => '',
  'kind' => '',
  'domain' => 'exp',
  'agendastatus' => 'none',
  'globalorder' => 0,
  'todo' => 120,
  'cluster' => '/AVE',
  'flags' => 8,
  'account' => '',
  'p_agenda_cpus' => -1,
  'p_agenda_priority' => -1,
  'retrysubjob' => -1,
  'path' => 'C:/Program Files/pfx/qube//types;C:/Program Files/pfx/jobtypes;C:/Program Files (x86)/pfx/jobtypes;//bluearc/render/qube/templates/jobtypes',
  'requirements' => '',
  'package' => bless( {
    'mayaExecutable' => 'C:\\Program Files\\Autodesk\\Maya2013\\bin\\mayabatch.exe',
    'defaultRenderGlobals.currentRenderer' => 'mentalRay',
    'project' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest',
    'scenefile' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\scenes\\rentest_v01.ma',
    'range' => '1-120',
    'renderDirectory' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\images',
    'rangeExecution' => 'individual',
    'frameCount' => '120'
  }, 'maya::Package' ),
  'timecomplete' => 'None',
  'notes' => '',
  'cpus' => 10
}, 'maya::MayaJob' );


==================== INIT BEGIN ====================
HOME:
QBDIR: C:\Program Files\pfx\qube\
MAYA_LOCATION:
MAYA_PLUG_IN_PATH: C:\Program Files\Pixar\RenderManStudio-3.0.2-maya2012\rmantree\bin
MAYA_SCRIPT_PATH:

Project Directory: //bluearc/render/Students/joeldpn/RenderTest
Render Directory:  //bluearc/render/Students/joeldpn/RenderTest/images
Scene File:        //bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma

$VAR1 = bless( {
  'mayaExecutable' => 'C:\\Program Files\\Autodesk\\Maya2013\\bin\\mayabatch.exe',
  'defaultRenderGlobals.currentRenderer' => 'mentalRay',
  'project' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest',
  'scenefile' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\scenes\\rentest_v01.ma',
  'range' => '1-120',
  'renderDirectory' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\images',
  'rangeExecution' => 'individual',
  'frameCount' => '120'
}, 'maya::Package' );

INFO: scenefile [//bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma] exists and is readable
Scene File Maya Version: 2013
INFO: Using maya executable specified in submission
  [C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe]
Maya Binary: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe
maya::UniversalMayaRenderJob::initialize
maya::PromptMayaJob::initialize
maya::PromptMayaJob::mayaSetupWorkspace
INFO: setting MAYA_LOCATION to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt
exception caught! at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 382.
ERROR: exception caught in maya::UniversalMayaRenderJob=HASH(0x4391b98)->run()
file not found: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 379

INFO: reporting status [failed] to supe: qb::reportjob('failed')
/qube/templates/jobtypes/maya/MelProcessor.pm line 379

INFO: reporting status [failed] to supe: qb::reportjob('failed')
)
maya::MelProcessor::DESTROY
maya::MelProcessor::finish

INFO: reporting status [failed] to supe: qb::reportjob('failed')
)
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
h
----------------------------------
-     Worker 1084.6
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.
----------------------------------
-     Worker 1084.7
----------------------------------

$VAR1 = bless( {
  'hosts' => '',
  'priority' => 9999,
  'retrywork' => -1,
  'timesubmit' => 'Jan 11, 2013 8:07:58',
  'flagsstring' => 'auto_mount',
  'omithosts' => '',
  'lastupdate' => 'Jan 11, 2013 8:08:04',
  'pid' => 1,
  'timeout' => -1,
  'localorder' => 0,
  'groups' => '',
  'user' => 'cniew',
  'id' => 1084,
  'prototype' => 'maya',
  'mailaddress' => '',
  'restrictions' => '',
  'hostorder' => '',
  'retrywork_delay' => 0,
  'omitgroups' => '',
  'reason' => '',
  'name' => 'AVE-mray2013-2013test',
  'data' => '(=(scenefile=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest\\\\scenes\\\\rentest_v01.ma)(defaultRenderGlobals.currentRenderer=mentalRay)(rangeExecution=individual)(frameCount=120)(mayaExecutable=C:\\\\Program Files\\\\Autodesk\\\\Maya2013\\\\bin\\\\mayabatch.exe)(project=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest)(range=1-120)(renderDirectory=\\\\\\\\bluearc\\\\render\\\\Students\\\\joeldpn\\\\RenderTest\\\\images))',
  'todotally' => {
    'complet' => 0,
    'killed' => 0,
    'running' => 0,
    'pending' => 0,
    'badlogin' => 0,
    'failed' => 0,
    'waiting' => 0,
    'unknown' => 0,
    'suspended' => 0,
    'blocked' => 0
  },
  'timestart' => 'Jan 11, 2013 8:08:01',
  'serverid' => 0,
  'agenda_timeout' => -1,
  'label' => 'qube1',
  'queue' => '',
  'pgrp' => 1084,
  'reservations' => 'host.processors=1+',
  'cpustally' => {
    'complet' => 0,
    'killed' => 0,
    'running' => 0,
    'pending' => 0,
    'badlogin' => 0,
    'failed' => 0,
    'waiting' => 0,
    'unknown' => 0,
    'suspended' => 0,
    'blocked' => 0
  },
  'status' => 'running',
  'subjobstatus' => 'pending',
  'dependency' => '',
  'kind' => '',
  'domain' => 'exp',
  'agendastatus' => 'none',
  'globalorder' => 0,
  'todo' => 120,
  'cluster' => '/AVE',
  'flags' => 8,
  'account' => '',
  'p_agenda_cpus' => -1,
  'p_agenda_priority' => -1,
  'retrysubjob' => -1,
  'path' => 'C:/Program Files/pfx/qube//types;C:/Program Files/pfx/jobtypes;C:/Program Files (x86)/pfx/jobtypes;//bluearc/render/qube/templates/jobtypes',
  'requirements' => '',
  'package' => bless( {
    'mayaExecutable' => 'C:\\Program Files\\Autodesk\\Maya2013\\bin\\mayabatch.exe',
    'defaultRenderGlobals.currentRenderer' => 'mentalRay',
    'project' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest',
    'scenefile' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\scenes\\rentest_v01.ma',
    'range' => '1-120',
    'renderDirectory' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\images',
    'rangeExecution' => 'individual',
    'frameCount' => '120'
  }, 'maya::Package' ),
  'timecomplete' => 'None',
  'notes' => '',
  'cpus' => 10
}, 'maya::MayaJob' );


==================== INIT BEGIN ====================
HOME:
QBDIR: C:\Program Files\pfx\qube\
MAYA_LOCATION:
MAYA_PLUG_IN_PATH: C:\Program Files\Pixar\RenderManStudio-3.0.2-maya2012\rmantree\bin
MAYA_SCRIPT_PATH:

Project Directory: //bluearc/render/Students/joeldpn/RenderTest
Render Directory:  //bluearc/render/Students/joeldpn/RenderTest/images
Scene File:        //bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma

$VAR1 = bless( {
  'mayaExecutable' => 'C:\\Program Files\\Autodesk\\Maya2013\\bin\\mayabatch.exe',
  'defaultRenderGlobals.currentRenderer' => 'mentalRay',
  'project' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest',
  'scenefile' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\scenes\\rentest_v01.ma',
  'range' => '1-120',
  'renderDirectory' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\images',
  'rangeExecution' => 'individual',
  'frameCount' => '120'
}, 'maya::Package' );

INFO: scenefile [//bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma] exists and is readable
Scene File Maya Version: 2013
INFO: Using maya executable specified in submission
  [C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe]
Maya Binary: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe
maya::UniversalMayaRenderJob::initialize
maya::PromptMayaJob::initialize
maya::PromptMayaJob::mayaSetupWorkspace
INFO: setting MAYA_LOCATION to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt
exception caught! at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 382.
ERROR: exception caught in maya::UniversalMayaRenderJob=HASH(0x41b4b20)->run()
file not found: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 379

INFO: reporting status [failed] to supe: qb::reportjob('failed')
/qube/templates/jobtypes/maya/MelProcessor.pm line 379

INFO: reporting status [failed] to supe: qb::reportjob('failed')
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
ish

INFO: reporting status [failed] to supe: qb::reportjob('failed')
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
ish
ish

----------------------------------
-     Worker 1084.8
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.

----------------------------------
-     Worker 1084.9
----------------------------------
'perl' is not recognized as an internal or external command,
operable program or batch file.

[jburk]

You need to install ActiveState Perl on all machines that report:

'perl' is not recognized as an internal or external command


You can find it here:

http://www.activestate.com/activeperl/downloads

Tip: select "don't install" for the Perl documentation to make the install finish much quicker.

You can test the perl installation on all workers by submitting a cmdline job with the "host_list" flag set that runs:

perl -v


If it prints out anything other than "not a recognized...", you're good.
 
Then, restart the worker service or reboot the machine once you've finished the Perl install.

[Chris Computing]

I have version 5.14.2 installed on all machines and it is still giving me this error. Though, on some of the machines it wasn't showing as a "path" in the PATH environmental variables in Windows.

Thanks!

[michael.graf]

were the machines rebooted after the perl install?

Sometimes Window's services will not grab a recent env change.

so if you are at a recently installed machine and the registry shows the PATH is updated but the service/worker is not reflecting this update, then a reboot is needed.

[Chris Computing]

Well, the Perl issue is gone, however I get this for ever single machine in our network: ==================== INIT BEGIN ====================
HOME:
QBDIR: C:\Program Files\pfx\qube\
MAYA_LOCATION:
MAYA_PLUG_IN_PATH:
MAYA_SCRIPT_PATH:

Project Directory: //bluearc/render/Students/joeldpn/RenderTest
Render Directory:  //bluearc/render/Students/joeldpn/RenderTest/images
Scene File:        //bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma

$VAR1 = bless( {
  'mayaExecutable' => 'C:\\Program Files\\Autodesk\\Maya2013\\bin\\mayabatch.exe',
  'defaultRenderGlobals.currentRenderer' => 'mentalRay',
  'project' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest',
  'scenefile' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\scenes\\rentest_v01.ma',
  'range' => '1-120',
  'renderDirectory' => '\\\\bluearc\\render\\Students\\joeldpn\\RenderTest\\images',
  'rangeExecution' => 'individual',
  'frameCount' => '120'
}, 'maya::Package' );

INFO: scenefile [//bluearc/render/Students/joeldpn/RenderTest/scenes/rentest_v01.ma] exists and is readable
Scene File Maya Version: 2013
INFO: Using maya executable specified in submission
  [C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe]
Maya Binary: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe
maya::UniversalMayaRenderJob::initialize
maya::PromptMayaJob::initialize
maya::PromptMayaJob::mayaSetupWorkspace
INFO: setting MAYA_LOCATION to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt
to C:\Program Files\Autodesk\Maya2013
launching: C:\Program Files\Autodesk\Maya2013\bin\mayabatch.exe -batch -prompt


INFO: timer of [300 secs] expired-- "nudging" maya for 'mel:' prompt at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 554.
ERROR: maximum nudge count

• reached on command []

timed out-- aborting! at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 569.
maya::MelProcessor::abort
rt

ERROR: exception caught in maya::UniversalMayaRenderJob=HASH(0x3085678)->run()
aborting at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 137.

INFO: reporting status [failed] to supe: qb::reportjob('failed')
')
maya::MelProcessor::DESTROY
maya::MelProcessor::finish
INFO: HARNESS=[IPC::Run=HASH(0x3086700)]
ERROR: did maya crash earlier? at //bluearc/render/qube/templates/jobtypes/maya/MelProcessor.pm line 106 during global destruction.
on.

=================  qube! - retry/requeue on Jan 14, 2013 12:13:52  ===================

[Chris Computing]

Wait, does the Qube proxy account have to be an administrator? It creates itself as a basic user account, and I'm wondering if the whole time out nudging issue is permissions based.


Also, would it be fine if I changed the password for that proxy account? as it is anyone could google it and find out the default password. Will that break anything if i change it.

[michael.graf]

the qube proxy should not be admin, but the username used as the proxy user should have permission/s to access the job folders.

we changed the proxy user to one of our service accounts so it would have proper permissions to the file servers.

[Chris Computing]

Strange, the only way I was able to make this work was to make that account an administrator account. How do i keep it a regular user account and give it permissions to access the areas it needs? And what area does it need to access on the PC's?

On a side note, at least not our school can render to all the machines.