Author Topic: stderr for work/agenda  (Read 10912 times)

vicken

  • Jr. Member
  • **
  • Posts: 8
stderr for work/agenda
« on: November 11, 2014, 07:44:43 PM »
Hi, is it possible to get the stderr log using the python api, for a specific work/agenda item? We'd like to be able to view the logs of a specific frame, back-to-back for each time it has been retried (similar to Qube wrangleview, when you select an item in the Frames/Work panel).

I was hoping for something like
qb.stderr("JobId:WorkId")

but it _seems_ like it's only possible to query based on at most subjob
qb.stderr("JobId:SubjobId")

or is there a different way to go about it that i should use?

BrianK

  • Hero Member
  • *****
  • Posts: 107
Re: stderr for work/agenda
« Reply #1 on: November 11, 2014, 08:41:17 PM »
Sadly, this operation is non-trivial.

Qube stores logs based on the instance, not the frame, so each instance has its own log, and in those log files are the frame logs.  Both WranglerView and ArtistView deliver frame logs by finding the instance that rendered the frame, getting that instance's log, then parsing the log to find the frame start and stop point.

That said, as luck would have it, I've written this feature as an example for someone in the past & am happy re-share here on the forum.  I haven't tested this in a while (and can't test it currently, due to the state of my machine), but I'm fairly certain it still works.  This particular python script is designed to be used from the command line like so:

qb_getFrameLog.py <job_id>:<frame_name>

so

qb_getFrameLog.py 1234:56

It shouldn't be too difficult to refactor into your own python scripts.

The jist of it is this:

1. Given a job:frame, it finds the frame object by querying the supervisor with qb.jobinfo with 'agenda' set to True.
2. Once the frame object is found, we can query it to find the instance that created it.
3. Once the instance is found, we pull the log data from the supe with qb.stdout/qb.stderr, giving the instance id as an argument.
4. Once we have the logs, we look for the token that signifies frame start/stop.  That token is "got work <job_id>:<frame_name>".
5. It then returns a list of two elements: the stdout and stderr logs (each as text, IIRC)

Code: [Select]
#!/usr/bin/env python

"""
Provides the function getFrameLog that will return the logs for a given frame.
"""


import os
import sys
import re
import time
import datetime
import mmap

try:
    import qb
except ImportError:
    api_path = "/Applications/pfx/qube/api/python/"
    if api_path not in sys.path:
        sys.path.insert(0,api_path)
    import qb


def getFrameLog(frame):
    """
    takes a frame (qb.Work) & returns the log.
    return comes in the form [stdout,stderr]
    """

    ret = []

    jobframe    = '%d:%s' % (frame['pid'],frame['name'])
    subjobframe = '%d.%d' % (frame['pid'],frame['subid'])

    out_logs = qb.stdout(subjobframe)[0]
    err_logs = qb.stderr(subjobframe)[0]

    out_map = out_logs['data']
    err_map = err_logs['data']

    iter_pat = re.compile('got work: \d+:(\S+)')
    stdout_iter = re.finditer(iter_pat, out_map)
    stderr_iter = re.finditer(iter_pat, err_map)

    self_pat = re.compile("%s$" % jobframe)


    for count,iterator in enumerate((stdout_iter,stderr_iter)):
        start = end = -1
        for i in iterator:
            if start >= 0:
                end = i.span()[0]
                break
            if re.search(self_pat,(out_map,err_map)[count][slice(*i.span())]):
                start = i.span()[0]
        if start >= 0:
            print "%s frame log start/stop for frame '%s' of job '%d.%d': %d/%d" % ({0:'stdout',1:'stderr'}[count],
                                                                                    frame['name'],
                                                                                    frame['pid'],
                                                                                    frame['subid'],
                                                                                    start,
                                                                                    end)
            line_start = max(0,(out_map,err_map)[count].rfind('\n',0,start))
            line_end = max(0,(out_map,err_map)[count].rfind('\n',0,end))
            ret.append((out_map,err_map)[count][line_start:line_end])
        else:
            ret.append('')


    return ret

def main(args):
    """
    Takes a list of args (probably from the command line) that look like
    job:frame (i.e. 1234:23) and returns the logs for that frame
    """
    for arg in args:
        try:
            job_id,frame_number = arg.split(':')
        except ValueError:
            continue

        # find the frame id based on the frame name(number)
        frame_number = int(frame_number)
        frame_number -= 1
        found_id = -1
        job = qb.jobinfo(id=job_id,agenda=True)[0]
        for fid,fname in ((frame['id'],frame.get('name')) for frame in job['agenda']):
            frange = map(int,fname.split('-'))
            if frame_number >= frange[0] and frame_number <= frange[-1]:
                found_id = fid
                break

        if found_id < 0:
            print >>sys.stderr, "Could not find frame number '%s'" % frame_number
            sys.exit(1)

        print getFrameLog(job['agenda'][found_id])
   
   

if __name__ == "__main__":
    main(sys.argv)