Custom backend: how to get 1 entry for every processed frame in qube supervisor?

[juanjux]

Hi,

I'm implementing a custom backend, similar to mayaprompt. It works, but I've the problem that in the Qube Supervisor "Agenda/Frames" tab there isn't a 1 to 1 correspondence between the entries in the GUI and the processed frames output, that is, if I've a work with 10 frames and a single worker (to simplify the example), for example the frame "1" in the tab shows as complete but without output (it just shows "requesting work for: XXXX.X"), same for the frame "2", then the output entry for the frame "3" shows the output of the first 3 frames, etc. It repeats the same pattern every three frames, that is, the entries for the frames 4 and 5 would again miss any output and the entry for the frame 6 would have the output of the frames 4, 5 and 6.

I'm using the way documented here and what I've seen mayaprompt uses (I'm using Python):

Code: [Select]

job = qb.jobobj()

jobstate = 'complete'

while True:

  agendaItem = qb.requestwork()
  if agendaItem['status'] == 'complete':
    break
  elif agendaItem['status'] == 'blocked':
    jobstate = 'blocked'
    break
  elif agendaItem['status'] == 'pending':
    jobstate = 'pending'
    break

  else:
    try:
      print 'BEGIN ITEM %s' % str(agendaItem['name'])
      # ...omitted, work with the job and job['package'] to generate the frame...
    except Exception, e:
      print 'WORKER: Exception: %s\n' % str(e)
      print_exc()
      agendaItem['status'] = 'failed'
      jobstate = 'failed'
   else:
      agendaItem['status'] = 'complete'
   qb.reportwork(agendaItem)
qb.reportjob(state)
    

Thanks,
Juanjo Alvarez
(Ilion Animation Studios Developer)

[jburk]

This has to do with the job's "agenda", or list of work to be performed at the time the job is built at the submitting host.

If you're working over a frame range, your agenda currently probably contains a single item with the frame range as a string: '1-5'  You have a list with a single item in it.

What you want to do is to generate a list of work, one per frame:

Code: [Select]

job = {}
fStart = 1
fEnd = 5
fStep = 1

job['agenda'] = []

# you can do it manually
for i in range(fStart, fEnd+1):
    agendaItem = {'name': i}
    job['agenda'].append( agendaItem )

# job['agenda'] is now
# [{'name': 1}, {'name': 2}, {'name': 3}, {'name': 4}, {'name': 5}]

# or you can use one of qb's gen* convenience functions:
import qb

# the same thing as the above iteration loop
job['agenda'] = qb.genframes( '%s-%s' % (fStart, fEnd))

# job['agenda'] is also now
# [{'name': 1}, {'name': 2}, {'name': 3}, {'name': 4}, {'name': 5}]

# divide the agenda into 2-frame "chunks"
job['agenda'] = qb.genchunks(2, '%s-%s' % (fStart, fEnd))

# job['agenda'] is now
# [Work({'name': '1-2'}), Work({'name': '3-4'}), Work({'name': '5'})]

# Or if you don't care about the chunksize, but want to split it up across a known number of items:
# split 73 frames evenly as possible across 4 items:
job['agenda'] = qb.genpartitions(4, '1-73')

# job['agenda'] is now
# [Work({'name': '1-19'}), Work({'name': '20-38'}), Work({'name': '39-57'}), Work({'name': '58-73'})]
   


[juanjux]

No, I'm already generating the agenda (using genframes). I've printed job['agenda'] and all seems to be Ok, but it still only process 3 frames in every third entry. This is a screencap of the Qube Supervisor for this work:



Of the frames on this image, the distribution of works really processed are (using a single worker):

1: None
2: None
3: 1,2,3
4: None
5: None
6: 4, 5, 6
7: None
8: None
9: 7, 8, 9

...etc...