Tip: How to write out a "csv" file of job data

[Scot Brew]

Since Qube has a python API to accesss the job and host information, that can be used to collect information and then write it out to a csv file.  Python lists and dictionaries are used to store that information, making it easy to access and then print with the standard csv python module.

Code: [Select]

% python
>>> import qb
>>> import csv
>>> jobs = qb.jobinfo()                                # Get a list of all the jobs.  See qb.jobinfo() docs for how to get a subset of the jobs
>>> keysList = jobs[0].keys()                       # Get a list of the keys to write out.  Alternatively specify your own list like ['id', 'name']
>>> f = open('/tmp/aaa.csv', 'w')                  # Open a csv file for writing
>>> c = csv.DictWriter(f, keysList)                 # Attach a csv dict writer to that file
>>> c.writerow(dict([(k,k) for k in keysList])) # Add a title line to the csv file
>>> c.writerows(jobs)                                   # Write out the job info for the specified fields
>>> f.close()                                                 # Close the file

[Scot Brew]

The time fields use a standard timestamp number (in seconds from standard epoch).  That number can be converted to a date/time by using the python datetime module.

Code: [Select]

% python
>>> import datetime
>>> import qb
>>> t = qb.jobinfo(id=1500)[0]['timestart']
>>> print t
1221804327
>>> d = datetime.datetime.fromtimestamp(t)
>>> print d
2008-09-18 23:05:27
To convert timestamps to this more readable form in-place within the job dicts, you can use the following code snippet:

Code: [Select]

import qb
import datetime
jobs = qb.jobinfo()
for j in jobs:
    for k in('timesubmit', 'timestart', 'timecomplete'):
        j[k] = datetime.datetime.fromtimestamp(j[k])

Result
Combining the datetime conversion code with the csv code produces:

Code: [Select]

import qb
import csv
import datetime

# Get a list of all the jobs.  See qb.jobinfo() docs for how to get a subset of the jobs
jobs = qb.jobinfo()   

# Convert all timestamps to datetime format
for j in jobs:
    for k in('timesubmit', 'timestart', 'timecomplete'):
        j[k] = datetime.datetime.fromtimestamp(j[k])

# Write out the csv file
keysList = jobs[0].keys()    # List of the keys to write out.  Alternative: ['id', 'name'], etc.
f = open('/tmp/aaa.csv', 'w')    # Open a csv file for writing
c = csv.DictWriter(f, keysList)   # Attach a csv dict writer to that file
c.writerow(dict([(k,k) for k in keysList])) # Add a title line to the csv file
c.writerows(jobs)    # Write out the job info for the specified fields
f.close()        # Close the file


[awells]

Would you happen to have any documentation on qb.jobinfo other than what's included in the dev manual and API reference that ship with Qube?

Also, I'm having a little trouble customizing the keyList to only include the wanted data. Listing data sets individually (keyList = ['id', 'name', 'etc', 'etc..']) gives me an error (dict contains fields not in fieldnames) so I'm assuming I'm misunderstanding your notes.

I was also curious as to whether or not it's possible to query individual parts of todotally and cpustally.  Let's say I just wanted the number of running jobs, for example.

Any other tips and suggestions you can offer, as always, are more than welcome.

Thanks Again.

[shinya]

Hi awells,

No, we don't have additional documentation on qb.jobinfo.
Is there anything in particular that you'd like to know about it?

I'm not sure what your issue is with keyList-- could you post the code
you're having trouble with? 

The todotally and cpustally come as is, with all information.
You can always just access the individual pieces of info you need after the query, as in:

Code: [Select]

print jobs[0]['todotally']['running']