dictionary to submit a maya job (new in qube)

[coclea]

Hi

The function qb.genFramces( agendaRange) does not seems to exists in qb loaded in maya 2012 on linux. Nevertheless I created my way an agenda with callback added.

I am trying to kick a job, that will launch subtasks, hoping each sub task will kick a cshell command:


      agenda = []      
      packets = int(10)
      nbJobs = float((end - start)/packets)
      if (nbJobs > int(nbJobs) ):
         nbJobs = int(nbJobs+1)
      agendaRange = '%s-%sx%s' % (int(0), int(nbJobs),int(1))
          listOfCallbacks = []
      for i in range( 0, nbJobs):
         work = {}
         work['name'] = '%s-%s'%(int(start+i*packets), int(  min( end, start + (i+1)*packets)))
         }
         callback = {}
         callback['triggers'] = 'complete-work-parentJob-%s' % work['name']
         callback['language'] = 'cshell'
         callback['code'] = ('/mnt/RT0/tools/SITE/software/Linux/maya_2012_SP2/bin/Render -renderer file ' + '-s ' + str( start + i*packets) + ' -e ' + str( min( end, start + (i+1)*packets-1))  + ' -rd '+ renderPath  + ' ' + scriptPath)
         listOfCallbacks.append( callback)   
         agenda.append(work)
      scriptName = maya.file( sn = True, shn = True, q = True)
      submitDict = {
           'name'      : 'maya '+ scriptName,
           'prototype' : 'cmdrange',
      'requirements':'host.os=linux',
      'priority': 100,
           'agenda' : agenda,
      'cpus' : 8,
      'callbacks' : listOfCallbacks,
      'package' : {
         'frameCount' : packets
                   }
             }

                listOfJobs = []
          listOfJobs.append(submitDict)
          listOfSubmittedJobs = qb.submit(listOfJobs)


It kicks callbacks, but none of them run. it says:
language : none

How shall I do??
I am writing a custom interface to kick the process.
I could not find any where a definition for agenda, package, jobDictionary..
I don't know what kind of portotypes are available : cdmline, cmdrange ....

Thanks for your help

[coclea]

Hi

So I have noticed that the allowed languages where :
['perl', 'python', 'tcl', 'qube', 'none', 'post', 'mail', 'dependency', 'auto_wrangling']

So I changed my code to submit python:
      agenda = []      
      packets = int(10)
      nbJobs = float((end - start)/packets)
      if (nbJobs > int(nbJobs) ):
         nbJobs = int(nbJobs+1)
      agendaRange = '%s-%sx%s' % (int(0), int(nbJobs),int(1))
          listOfCallbacks = []
      for i in range( 0, nbJobs):
         work = {}
         work['name'] = '%s-%s'%(int(start+i*packets), int(  min( end, start + (i+1)*packets)))
         #work['name'] = '%s'%(int(start+i*packets))

         callback = {}
         callback['triggers'] = 'complete-work-parentJob-%s' % work['name']
         callback['language'] = 'python'
         cmd = 'import os; os.system(\'/mnt/RT0/tools/SITE/software/Linux/maya_2012_SP2/bin/Render -renderer file -s ' + str( start + i*packets) + ' -e ' + str( min( end, start + (i+1)*packets-1))  + ' -rd '+ renderPath  + ' ' + scriptPath + '\');'
         callback['code'] = cmd
         listOfCallbacks.append( callback)   
         agenda.append(work)
      
      scriptName = maya.file( sn = True, shn = True, q = True)
      submitDict = {
           'name'      : 'maya '+ scriptName,
           'prototype' : 'cmdrange',
      'requirements':'host.os=linux',
      'priority': 100,
           'agenda' : agenda,
      'cpus' : 8,
      'callbacks' : listOfCallbacks,
      'package' : {
         'frameCount' : packets
                   }
             }
        listOfJobs = []
          listOfJobs.append(submitDict)
          listOfSubmittedJobs = qb.submit(listOfJobs)

So I get 18 callbacks with python language and my python os.system command is right. But none of the subprocesses starts.

They are indicated as ready.

Any idea?

Thanks

[jburk]

Please submit a ticket to our helpdesk by sending mail to support@pipelinefx.com.  If you have never interacted with our helpdesk before, the system will send you an account verification email.  You'll need to reply to this email, otherwise your first support case will sit in a "suspended" state until such time that you do.

When you do submit the email, could you attach the script which contains the code you're trying to run?

But, it may be very much simpler than this.  I don't believe callbacks are what you want to use, since these are primarily used for defining dependencies between job, not executing the jobs themselves.

as well, what gets returned when you run "dir(qb)" in a python scriptEditor window in Maya.  And what does "print qb.__file__" say?  Wondering where it's finding the qb module.

Try modifying your submission script so that it looks more like this:

Code: [Select]

    for i in range( 0, nbJobs):
         work = {}
         work['name'] = '%s-%s' % (int(start+i*packets), int( min( end, start + (i+1)*packets)))
         agenda.append(work)
     
      scriptName = maya.file( sn = True, shn = True, q = True)
      submitDict = {
            'name'      : 'maya '+ scriptName,
            'prototype' : 'cmdrange',
            'requirements':'host.os=linux',
            'priority': 100,
            'agenda' : agenda,
            'cpus' : 8,
            'callbacks' : listOfCallbacks,
            'package' : {
                'frameCount' : packets,
                'range': '1-%s' % nbJobs,
                'cmdline': '/mnt/RT0/tools/SITE/software/Linux/maya_2012_SP2/bin/Render -renderer file -s QB_FRAME_START -e QB_FRAME_END -b QB_FRAME_STEP -rd %s %s' % (renderPath, scriptPath)
           }
     }


The QB_* tokens in the command-line will be replaced with the appropriate values by the cmdrange jobtype when the job runs on the worker.

To see these in action, try submitting the following jobs:

Code: [Select]

import qb

jobA = {
    'prototype': 'cmdrange',
    'name': 'jobA',
    'package': {
        'padding': 2,
        'cmdline': 'echo fNum:QB_FRAME_NUMBER fStart:QB_FRAME_START fEnd:QB_FRAME_END fStep:QB_FRAME_STEP fRange:QB_FRAME_RANGE'
    },
    'agenda': []
}

for i in range(1,10):
    work = {'name': i}
    jobA['agenda'].append(work)


jobB = {
    'prototype': 'cmdrange',
    'name': 'jobB',
    'package': {
        'padding': 4,
        'cmdline': 'echo fNum:QB_FRAME_NUMBER fStart:QB_FRAME_START fEnd:QB_FRAME_END fStep:QB_FRAME_STEP fRange:QB_FRAME_RANGE'
    },
    'agenda': []
}

step = 5
for i in range(1,30, step):
    work = {'name': '%s-%sx%s' % (i, i+step-1, step)  }
    jobB['agenda'].append(work)


for j in qb.submit([jobA, jobB]):
    print '%(name)s: %(id)s' % j

You'll find the first will print out the following for frame 8:

fNum:08 fStart:08 fEnd:08 fStep:01 fRange:8


the second job will print out the following for chunk 11-15x5:

fNum:0011 fStart:0011 fEnd:0015 fStep:0005 fRange:11-15x5

[coclea]

Hi

That's definitly helped.

I will try again with callback as I would like to have an interface that kick a cache, assign the cache in the maya scene tahn render it, all in one job overnight! So I guess callback is the right function for that.

Anyway, I have a script running now. Just the strings that I am getting from a maya python interface are not working straight away. I have to cast them otherwise the cmdline is not sent.
I am using str(scriptPath) and not simply scriptPath.

      cmdLine = ('/mnt/RT0/tools/SITE/software/Linux/maya_2012_SP2/bin/Render -r file -rd %s -s QB_FRAME_START -e QB_FRAME_END %s'%(str(renderPath), str(scriptPath)))

      submitDict = {
           'name'      : 'maya '+ scriptName,
           'prototype' : 'cmdrange',
      'requirements':'host.os=linux',
      'priority': priority,
           'agenda' : agenda,
      'cpus' : cpus,
      'groups' : 'mantra',
      'package' : {
              'cmdline': cmdLine,
         'frameCount' : packets,
         'range' :  '1-%s' % (int(nbJobs)),
         'script' : baseName,
         'regex_outputPaths ': 'Writing (.*) took [0-9.\-]+ seconds   ',
         'executable' : "/mnt/RT0/tools/SITE/software/Linux/maya_2012_SP2/bin/Render",
                   }
             }

The agenda command is genframes and not genFrames (so it is working!!)
So I get
print qb.__file__
/usr/local/pfx/qube/api/python/qb/__init__.pyc
dir(qb)
# Result: ['Callback', 'Host', 'Job', 'QBObject', 'QB_API_BINARY', 'QB_API_XML', 'QB_CLIENT_DEFAULT_CONF', 'QB_SUPERVISOR_CONFIG_DEFAULT_LICENSE_FILE', 'QB_SUPERVISOR_CONFIG_DEFAULT_LOGFILE', 'QB_SUPERVISOR_CONFIG_DEFAULT_LOGPATH', 'QB_SUPERVISOR_CONFIG_DEFAULT_WORKER_CONFIGFILE', 'QB_TIME_EPOCH_OFFSET', 'QB_WORKER_CONFIG_DEFAULT_LOGFILE', 'Subjob', 'Work', '__builtins__', '__doc__', '__docformat__', '__dontlookatme__', '__file__', '__name__', '__package__', '__path__', '_assignment', '_init_assign', '_qb', '_qb26', '_setjob', 'archivejob', 'binarySort', 'block', 'blockwork', 'bottom', 'checkpassword', 'complete', 'completework', 'convertpath', 'currenttime', 'deleteworkerproperties', 'deleteworkerresources', 'encryptpassword', 'error', 'genchunks', 'genframes', 'genpartitions', 'getlogpath', 'getresources', 'getsupervisor', 'gettimeout', 'getusers', 'hist', 'hostinfo', 'hostorder', 'interrupt', 'jobconfig', 'jobid', 'jobinfo', 'jobobj', 'joborder', 'jobtypeavailable', 'kill', 'killwork', 'localconfig', 'migrate', 'modify', 'os', 'ping', 'preempt', 'qbPythonVer', 'qbadmin_reconfigureworkers', 'rangechunk', 'rangepartition', 'rangesplit', 're', 'recoverjob', 'remove', 'reportjob', 'reportwork', 'requestwork', 'requeue', 'requeuework', 'resume', 'retire', 'retirework', 'retry', 'retrywork', 'setlogpath', 'setsupervisor', 'settimeout', 'setusers', 'shove', 'stats', 'stderr', 'stdout', 'subid', 'submit', 'supervisorconfig', 'suspend', 'sys', 'top', 'unblock', 'unblockwork', 'updatelocalconfig', 'updatepassword', 'updateresources', 'updateworkerconfig', 'updateworkerproperties', 'updateworkerresources', 'version', 'waitfor', 'warnings', 'workblock', 'workcomplete', 'workerconfig', 'workerlock', 'workerpathmap', 'workerping', 'workkill', 'workrequeue', 'workretry', 'workunblock'] #


Thanks a lot

[jburk]

Yes, as you've noticed, all strings in Maya are unicode, but the qb module doesn't like unicode, and will reject a job that contains unicode strings. When you cast the paths with str(), you're converting them to straight ASCII.

This is something that I admittedly need to determine why it's this way, I'll have to check and see if the MySQL database tables currently will support unicode.