Author Topic: Tip: How to write out a "csv" file of job data  (Read 9853 times)

Scot Brew

  • Hero Member
  • *****
  • Posts: 272
    • PipelineFX
Tip: How to write out a "csv" file of job data
« on: April 27, 2009, 06:45:50 PM »
Since Qube has a python API to accesss the job and host information, that can be used to collect information and then write it out to a csv file.  Python lists and dictionaries are used to store that information, making it easy to access and then print with the standard csv python module.

Code: [Select]
% python
>>> import qb
>>> import csv
>>> jobs = qb.jobinfo()                                # Get a list of all the jobs.  See qb.jobinfo() docs for how to get a subset of the jobs
>>> keysList = jobs[0].keys()                       # Get a list of the keys to write out.  Alternatively specify your own list like ['id', 'name']
>>> f = open('/tmp/aaa.csv', 'w')                  # Open a csv file for writing
>>> c = csv.DictWriter(f, keysList)                 # Attach a csv dict writer to that file
>>> c.writerow(dict([(k,k) for k in keysList])) # Add a title line to the csv file
>>> c.writerows(jobs)                                   # Write out the job info for the specified fields
>>> f.close()                                                 # Close the file

Scot Brew

  • Hero Member
  • *****
  • Posts: 272
    • PipelineFX
Re: Tip: How to write out a "csv" file of job data
« Reply #1 on: April 27, 2009, 08:16:32 PM »
The time fields use a standard timestamp number (in seconds from standard epoch).  That number can be converted to a date/time by using the python datetime module.

Code: [Select]
% python
>>> import datetime
>>> import qb
>>> t = qb.jobinfo(id=1500)[0]['timestart']
>>> print t
1221804327
>>> d = datetime.datetime.fromtimestamp(t)
>>> print d
2008-09-18 23:05:27

To convert timestamps to this more readable form in-place within the job dicts, you can use the following code snippet:

Code: [Select]
import qb
import datetime
jobs = qb.jobinfo()
for j in jobs:
    for k in('timesubmit', 'timestart', 'timecomplete'):
        j[k] = datetime.datetime.fromtimestamp(j[k])

Result
Combining the datetime conversion code with the csv code produces:
Code: [Select]
import qb
import csv
import datetime

# Get a list of all the jobs.  See qb.jobinfo() docs for how to get a subset of the jobs
jobs = qb.jobinfo()   

# Convert all timestamps to datetime format
for j in jobs:
    for k in('timesubmit', 'timestart', 'timecomplete'):
        j[k] = datetime.datetime.fromtimestamp(j[k])

# Write out the csv file
keysList = jobs[0].keys()    # List of the keys to write out.  Alternative: ['id', 'name'], etc.
f = open('/tmp/aaa.csv', 'w')    # Open a csv file for writing
c = csv.DictWriter(f, keysList)   # Attach a csv dict writer to that file
c.writerow(dict([(k,k) for k in keysList])) # Add a title line to the csv file
c.writerows(jobs)    # Write out the job info for the specified fields
f.close()        # Close the file

awells

  • Jr. Member
  • **
  • Posts: 8
Re: Tip: How to write out a "csv" file of job data
« Reply #2 on: April 30, 2009, 01:27:39 AM »
Would you happen to have any documentation on qb.jobinfo other than what's included in the dev manual and API reference that ship with Qube?

Also, I'm having a little trouble customizing the keyList to only include the wanted data. Listing data sets individually (keyList = ['id', 'name', 'etc', 'etc..']) gives me an error (dict contains fields not in fieldnames) so I'm assuming I'm misunderstanding your notes.

I was also curious as to whether or not it's possible to query individual parts of todotally and cpustally.  Let's say I just wanted the number of running jobs, for example.

Any other tips and suggestions you can offer, as always, are more than welcome.

Thanks Again.




shinya

  • Administrator
  • *****
  • Posts: 232
Re: Tip: How to write out a "csv" file of job data
« Reply #3 on: May 16, 2009, 01:30:10 AM »
Hi awells,

No, we don't have additional documentation on qb.jobinfo.
Is there anything in particular that you'd like to know about it?

I'm not sure what your issue is with keyList-- could you post the code
you're having trouble with? 

The todotally and cpustally come as is, with all information.
You can always just access the individual pieces of info you need after the query, as in:
Code: [Select]
print jobs[0]['todotally']['running']