Qube job logs

[PeterW]

We're writing some scripts to get statistics on various elements of our farm and have run into some issues getting data out of Qube's logs.  We've used  'qbstats' and 'qbsummary' to get some info but they aren't sufficient for what we'd like so we've written a Python script to use qb.jobinfo() to start grabbing the data we need.  The problem we've run into is partially related to the Qube GUI - we've had almost 20,000 jobs submitted to the farm over the last few months and the GUI loads increasingly slowly if old jobs are left behind.  qb.jobinfo() doesn't return any information on jobs that've been deleted from the GUI even though their logs still exist on the supervisor.

Is there any way to access those logs through the Python API?  Is there a better way of removing old job logs from the Qube GUI so that the logs are still accessible?

[Scot Brew]

There is no current way through the Python API to access the logs of removed jobs. However, the data-mining of historical job info (even for removed or archived jobs) is one of the areas we are looking to further develop in future versions of Qube.

At the present you will have to go out manually to the directory on the Supervisor where the job log files are stored. Alternatively, the location of those job logs can be configured in the qb.conf of the Supervisor to be stored on a network filesystem. One could then scan those job log directories directly with python or any other scripting language.